Night thoughts

How a Laptop Learned to Run a Giant

Haithem Slimi — Sun, 04 Oct 2026 10:27:15 GMT

New to the jargon? Start with the glossary →

A 70-billion-parameter model is roughly 140 GB of data. A mid-range gaming laptop has between 8 and 16 GB of video memory.

That's like trying to pour a swimming pool into a teacup. It shouldn't fit. So how is it running?

The answer is that we stopped trying to force the whole model into memory at once. Instead, we shrink it, we share the load, and we only use the parts we actually need. Here are the three tricks that make it possible.

Trick 1: Shrinking the Model

The first problem is pure size. You can't fit 140 GB into an 8 GB space. So we make the model smaller.

Quantization is the biggest lever. Think of it like a high-resolution photo. The original model uses 16-bit precision (called FP16), where every single parameter takes up 2 bytes of memory. That's the "high-res" version. Quantization drops that down to 8-bit (INT8), then 4-bit (INT4). At INT4, every parameter takes just 0.5 bytes.

That 140 GB model suddenly drops to around 35 GB. You lose a little bit of quality—the photo gets slightly blurrier—but you save an enormous amount of memory.

Pruning is the second lever. Researchers look at the model's billions of parameters and find the ones that aren't pulling their weight. They zero them out, effectively deleting them. It's like trimming dead branches off a tree so the rest can breathe.

Knowledge distillation is the third. Instead of compressing a big model, you train a small model to copy a big one. The small model learns from the big model's answers and reasoning, like an apprentice learning from a master. You end up with a model a fraction of the size, carrying a surprising amount of the original's capability.

Trick 2: Sharing the Load

Shrinking helps, but a 35 GB compressed model still doesn't fit comfortably into 16 GB of video memory. So we use everything the computer has.

Unified memory changed the game on Apple Silicon. In a traditional PC, the CPU has its own RAM, and the GPU has its own VRAM. Moving data between them is slow. Apple's M-series chips use a single pool of memory shared by both the CPU and the GPU. A MacBook Pro with 128 GB of unified memory can hold a ~65B model comfortably—no copying required.

CPU offloading is the PC equivalent. Instead of demanding that the entire model live on the graphics card, tools like llama.cpp split the model up. The "hot" layers that are needed right now stay on the GPU. The rest sit in system RAM, and the CPU handles them.

Is it slower than pure GPU inference? Yes. But it's a thousand times better than not running at all.

Trick 3: Doing Less Work

The final trick is architectural. Modern models are simply smarter about how they use themselves.

Mixture of Experts (MoE) is the biggest shift. Instead of a monolithic model that uses all 70 billion parameters for every single question, an MoE model is made of many smaller specialist teams. When a question comes in, a router sends it to just a few of those experts—maybe 8 to 10 billion parameters' worth. The model is huge on paper, but computationally cheap in practice. Examples like Mixtral, Qwen2.5-MoE, and Grok use this to deliver massive capability without massive compute costs.

Sparse attention reduces the memory and compute required for long contexts. Instead of forcing the model to look at every single word it has ever seen, sparse attention focuses only on the relevant parts. It cuts the math down drastically.

And then there's just better design. Models like Llama 3 and Qwen 2.5 are simply more parameter-efficient than their predecessors. A smaller model today beats a bigger model from two years ago.

The Reality Check

So what does this actually look like on your desk?

If you have a Mac Mini (M-series) with 32 GB of RAM, you can run Qwen2.5-14B or Llama-3-8B comfortably at INT4. You might even squeeze in a 32B model if you're aggressive with quantization and don't have much else open.

If you have a MacBook Pro with 64 GB or 128 GB of unified memory, the world opens up. You can run a 70B model at INT4. It's slow—roughly 5-10 tokens per second—but it runs, completely offline, no API calls needed. We're talking batch processing and careful prompting, not instant chat.

There is a tradeoff. Larger models mean slower inference, more heat, and louder fans. But here's the part that surprises most people.

Training an AI model requires massive, sustained compute. It's like running a marathon at sprint speed for weeks. But inference—just using the model to get an answer—doesn't need that same intensity. You're reading a huge matrix multiplication table, not calculating it from scratch.

That's why a laptop can do this. It's not performing miracles. It's just reading the answer, one compressed piece at a time.

Next in the series: The New Shape of AI Models →

New to the jargon? Every term in this series has a plain-English explanation in the glossary.

Bigger Isn't Always Smarter

Haithem Slimi — Sun, 04 Oct 2026 09:33:44 GMT

New to the jargon? Start with the plain-English glossary.

For years, the rule was simple: bigger AI meant smarter AI.

More parameters, more brainpower. Every new model was just the last one, scaled up. A 70-billion-parameter model was smarter than a 7-billion one, the way a bigger library holds more books. And if you wanted to run one of those giants, you needed a data center.

That rule just broke.

A 70-billion-parameter model now runs on a 4GB graphics card — the kind you'd find in a mid-range gaming laptop. Slowly, yes. But it runs. Under the old rule, that shouldn't be possible.

To understand why it matters, it helps to see where the old rule came from — and why it was right for so long.

The rule that worked

In 2020, researchers noticed something remarkable. As they made AI models bigger, the models got better in a way that was almost predictable. Double the size, and performance improved by a measurable amount. Double it again, and it improved again.

They called this "scaling laws." And for a few years, scaling laws were the closest thing AI had to a recipe.

The recipe went something like this:

More parameters: better.
More training data: better.
More compute: better.

If you wanted a smarter model, you made a bigger one.

And it worked. Models went from millions of parameters to billions, then to hundreds of billions. Each generation outclassed the last. The rule seemed less like a pattern and more like a law of nature.

Let's be fair: the old rule wasn't wrong. It was incomplete. For most of a decade, scaling up really did work. The problem wasn't that the rule failed — it's that we mistook it for the whole story.

Because something else was happening underneath.

The crack

In 2022, a paper called Chinchilla made a quiet but important point. Many of the biggest models weren't just big — they were undertrained. They had billions of parameters but hadn't seen enough data to use them well.

The implication was uncomfortable: a smaller model, trained on more data, could beat a bigger one trained on less.

At first, this was a footnote. Then it became a pattern. Small models started showing up on leaderboards next to giants. A well-trained 7-billion-parameter model could hold its own against one three times its size. In some cases, it beat them outright.

The old rule had a hole in it. And once you saw the hole, you couldn't unsee it.

Why the rule broke

Four things changed at roughly the same time. None of them alone explains the shift, but together they rewrote the rule.

Better data. For years, "more data" meant "scrape more of the internet." Then researchers started curating it — filtering, cleaning, and in some cases generating high-quality training data on purpose. A model fed clean, dense, well-chosen examples learns faster than one fed a mountain of noise. Quality started to matter as much as quantity.

Better teaching. Techniques like distillation let a small model learn from a big one. Instead of training from scratch, the small model copies the big model's answers and reasoning, the way an apprentice learns by watching a master. The result is a model a fraction of the size, with a surprising amount of the capability.

Smarter designs. The biggest shift might be architectural. Mixture-of-experts models are built from many specialist sub-models. When a question comes in, only a few of them wake up to answer. On paper, the model is enormous. In practice, only a slice of it runs at any given moment. Huge on paper, cheap in practice.

Better hardware. New chips, new number formats, and new software tricks let small machines hold more of the model at once. We'll dig into this in the next essay, but the short version is: the hardware caught up to the software's ambitions.

Sources: Official model cards, technical reports, and papers from Meta, Mistral AI, Alibaba Cloud, IBM, and DeepSeek (2023-2024). Scores are approximate and rounded.

Each of these forces pushed in the same direction: capability stopped being tied to raw size. A smaller model, designed and trained well, could do what used to require something much bigger.

What "smart" actually means

Here's the part that's easy to miss. "Smarter" was never a single number.

When we say a model is smart, we usually mean it's good at a specific set of tasks: answering questions, writing code, summarizing documents, reasoning through a math problem. But no model is good at everything. A model that writes beautiful prose might fail at arithmetic. A model that codes brilliantly might be hopeless at creative writing.

Benchmarks try to measure this, but they're imperfect. They get gamed. They get outdated. A model that tops one leaderboard can look mediocre on another.

So when we say "bigger isn't smarter," we're not saying size doesn't matter. We're saying size is one dimension of a model that has many dimensions — and that the other dimensions have gotten much more interesting.

The new question

If size isn't the measure, what is?

The honest answer is: it depends on what you need. For a chatbot that responds instantly, speed matters most. For a background process that runs overnight, speed matters less than cost. For a medical application, privacy might matter more than either.

The old question was "how big is it?" The new question is closer to: "how useful is it, how cheap is it, and how fast can it run — for what I'm actually doing?"

That question is harder to answer. But it's a much better one.

And it's the reason a 70-billion-parameter model can now live on a laptop.

That story — the how — is the next essay.

Plain English Glossary: AI Terms for the Rest of Us

Haithem Slimi — Sun, 04 Oct 2026 02:21:12 GMT

The Basics

Term	What it actually means
Parameter	A dial the model tuned while learning. Billions of them. Think of them as the model's settings.
Model size (e.g., 70B)	How many dials the model has. 70B = 70 billion. Bigger isn't automatically smarter.
Quantization	Storing the model in lower resolution, like a compressed photo. Slightly blurrier, much smaller.
FP16 / INT8 / INT4	The "resolution" of the model. FP16 is high-res (2 bytes per parameter). INT4 is compressed (0.5 bytes each). Lower numbers mean smaller files but slightly lower quality.
Inference	Running the model to get an answer. The moment you hit "send."
Training	The long, expensive process of teaching the model. Happens once, before you ever use it.
Tokens	Chunks of text the model reads and writes. Roughly ¾ of a word each.
Tokens per second	How fast the AI writes. 1–2/sec is a crawl. 40+/sec feels instant.
Time-to-first-token	How long you wait before the AI starts replying. The "thinking" pause.
VRAM	Special memory on a graphics card. The model has to fit here to run fast.
RAM	Your computer's regular memory. More RAM = you can hold more of the model at once.
KV cache	The model's short-term memory of your conversation so far. It grows as you chat.

The Runner-Ups

Term	What it actually means
MoE (Mixture of Experts)	A model made of specialist teams. Only a few wake up per question, so it's cheaper to run.
Active vs. total parameters	How many dials are used per answer vs. how many exist. A huge model can still be cheap.
Dense model	A model that uses all its dials for every answer. Simple but expensive.
Sparse model	A model that only uses part of itself per answer. Cheaper, often just as good.
Pruning	Trimming the model. Removing weights that aren't pulling their weight (setting them to zero) so the model takes up less space and runs faster.
Distillation	Teaching a small model to copy a big one. Like a student learning from a professor.
Sparse attention	A shortcut for long conversations. Instead of looking at every word it has ever seen, the model focuses only on the parts that matter right now.
Speculative decoding	Guessing ahead to speed up answers. If the guess is right, you save time.
Test-time compute	Letting the model think longer before answering. Smarter, but slower.
NPU	A chip built specifically for AI. Like a GPU, but for math instead of graphics.
Layer streaming / offloading	Keeping most of the model on disk, pulling in only what's needed, one piece at a time.
NVFP4 / FP8	Extra-compact number formats. Save memory, keep most of the quality.
Local inference	Running the model on your own machine. Private, offline, no subscription.
Cloud inference	Running the model on someone else's servers. Faster, stronger, but your data leaves home.
Hybrid routing	Small model handles easy questions; cloud handles the hard ones. The best of both.

Going Deeper (Optional)

These come up less often, but you might see them in the wild.

Term	What it actually means
Fine-tuning	Teaching an already-trained model a specific skill.
RAG (Retrieval-Augmented Generation)	Letting the model look things up before answering.
Context window	How much text the model can "hold in mind" at once.
Benchmark	A standard test used to compare models. Useful, but imperfect.
Scaling laws	The old rule that bigger models get predictably better. Now more nuanced.
FLOPs	A measure of raw computing work. Like counting how many math problems a chip can do.
Unified memory	A design where the CPU and GPU share the same pool of memory. Apple's approach.

This glossary is part of the Big Models, Small Machines series.

The Qwen Uncensored Model: How "Brain Surgery" Removed AI Safety Rules

Haithem Slimi — Sat, 03 Oct 2026 15:04:31 GMT

Imagine downloading an AI onto your laptop. It is smart, fast, and can answer almost any question. But it is missing one big thing: the word "no."

This is the reality of a new model called Qwen 3.8 27B Uncensored. Normally, AI companies build safety rules into their models. If you ask a standard AI how to make a dangerous weapon or write a computer virus, it will refuse. It will say, "I can't help with that."

But this version of Qwen doesn't do that. It will answer. It will not warn you. It will just give you the information you asked for.

This is not a mistake or a bug. It was done on purpose using a method called abliteration. It is a way to permanently remove an AI's ability to refuse a request. And because the code and the model are now public, the genie is out of the bottle.

The "Brain Surgery" Explained

To understand how this happened, we have to look at how AI makes decisions.

Think of an AI model like a giant, complex brain made of millions of connections. When you ask it a question, a signal travels through those connections. Normally, AI companies train their models to have a safety feature. We can call this the "refusal button."

If you ask a standard AI a dangerous question, the signal hits that button. The AI stops, and out comes a refusal: "I can't help with that."

Abliteration is a mathematical trick that finds that exact refusal button and deletes it.

It is not just telling the AI to ignore its rules. The button is physically gone. The math that creates the refusal is cut out of the model's code. The AI literally loses the ability to say no.

The most surprising part is that this doesn't require retraining the AI. It's not like teaching a dog a new trick. It's more like taking a car and removing the brakes. The car still drives perfectly fine. It's still fast and powerful. But it simply cannot stop anymore.

This is exactly what happened to the Qwen model. The creators took the original, safety-trained Qwen 3.8 27B and performed this "brain surgery." The result is a model that is just as smart and just as capable as the original, but completely unable to refuse a request.

The Qwen Model: Anyone Can Run It

So, what exactly is this Qwen model?

Qwen is a family of AI models created by the Chinese tech company Alibaba. They released the original version as "open-source," which means they shared the underlying code with the public. Anyone can download it, use it, or change it.

This is where the problem starts. Because the code is public, anyone with a decent computer and some technical knowledge can take the Qwen model and perform the "brain surgery" we just talked about. That is exactly what happened here.

The result is the Qwen 3.8 27B Uncensored model.

What makes this so concerning is how easy it is to run. You do not need a giant data center or a supercomputer. According to the model's documentation, it can run on:

A Mac: You need about 13.5 GB of free memory. If you have a newer Mac with an Apple Silicon chip, it runs even faster.
A Windows PC: You need a good graphics card (GPU) with about 16.8 GB of memory.

This means a normal person can download this model onto their laptop. They can run it completely offline. No internet connection is needed. No company is watching what they ask. It is a powerful, completely unrestricted AI, sitting right on someone's desk.

The Genie Out of the Bottle

This brings us to the biggest problem of all: we cannot stop it.

Once the mathematical method for abliteration was shared online, it was over. You cannot delete a math formula from the internet. You cannot un-share code. The method is out there, and it works.

We call this the "genie out of the bottle" problem.

In the past, if a company like OpenAI or Google made a dangerous AI, they could just shut down their servers. They could stop people from using it. But you cannot shut down a file that has been downloaded to a million different laptops around the world.

There is no central server to ban. There is no company to sue. The model is just a file, and it is free.

The Qwen Uncensored model proves that AI safety rules are not permanent. They are just a thin layer that can be peeled away. And now that everyone knows how to peel it away, they can do it to almost any other open-source AI model in the future.

The Bigger Picture

It is important to understand why people do this.

Some people argue that standard AI models are too strict. They refuse to help with harmless things, like writing a dark fiction story or asking about medical symptoms. They want an AI that is free from corporate rules. They believe in total privacy and total freedom.

But there is a serious downside.

A tool that cannot say "no" is a danger to everyone. A bad actor can use this Qwen model to write convincing scams, create computer viruses, or plan harmful acts. There are no guardrails. The AI will not warn the user. It will not alert the authorities. It will simply help.

We are now living in a world where powerful AI has no safety net. The genie is out of the bottle, and it is not going back in. The question is no longer whether we can stop it. The real question is: how do we live in a world where anyone can access a brilliant, obedient, and completely amoral digital assistant?

Why AI Agents Need Governance—and How Qodo Provides It

Haithem Slimi — Sat, 03 Oct 2026 02:20:50 GMT

AI agents can now write code. They can open pull requests, fix bugs, and add features at a speed humans cannot match. They do not get tired. That sounds like a big win for software teams.

But speed alone is not enough. If many agents write code at the same time, teams can lose control. Code may not follow company rules. Reviews may pile up. No one may see the full picture. A small change in one place can break something far away. A missed review can become a security problem. This is why control and governance are necessary.

Control means setting clear rules. Governance means making sure those rules are followed and that people can see what is happening. With AI agents, this is hard because work is spread across many repositories and pull requests. Teams need a way to guide agents, review their work, and understand the impact of their changes.

What Is Qodo?

Qodo is a platform that provides a quality and governance layer for software built with AI agents. The company was founded in 2022 by Itamar Friedman and Dedy Kredo. It was originally called CodiumAI and rebranded to Qodo in 2024 as the platform grew beyond just test generation. The founders have deep technical backgrounds: Friedman previously founded Visualead, which was acquired by Alibaba, and led machine vision work at Alibaba. [1]

The Team and Backing Behind Qodo

Qodo is backed by serious investors and advisors. The company has raised a total of $120 million, including a $70 million Series B round in March 2026. The round was led by Qumra Capital, with participation from Square Peg, Susa Ventures, TLV Partners, Vine Ventures, and others. Individual investors include Peter Welinder, VP of Product at OpenAI, and Clara Shih, VP of AI at Meta. [2]

The advisory board includes leaders from OpenAI, Meta, Shopify, and Snyk. This matters because it shows that people who understand both AI and enterprise software believe in what Qodo is building. The company is headquartered in New York. [3]

Why Qodo Is Promising

Qodo is not just another code review tool. It focuses on a problem that most tools ignore: understanding how a code change fits into the whole system. Most AI review tools look at what changed in a single pull request. Qodo looks at how that change affects the entire codebase, considering organizational standards, historical decisions, and risk tolerance.

The company’s CEO, Itamar Friedman, explained the thinking behind this: “Code generation companies are largely built around LLMs. But for code quality and governance, LLMs alone aren’t enough. Quality is subjective. It depends on organizational standards, past decisions, and tribal knowledge.” [4] This insight is the core of why Qodo exists.

The results speak for themselves. Qodo has an F1 score of 64.3% on the Code Review Bench, catching real problems at nearly twice the rate of others, including Claude. It is also ranked #1 by Gartner for code understanding in the Critical Capabilities for AI Assistants Report. Enterprises use it to catch an average of 800 bugs per month. [5]

How Qodo 3.0 Solves the Governance Problem

Qodo 3.0 is the latest version of the platform, released in October 2026. It adds new features designed specifically for governing AI-generated code at scale. Here is how it works:

PR Triage groups related pull requests across repositories into “work packages.” It scores them by impact, priority, and SLA. A reviewer can claim a package, so the same work is not reviewed twice. This directly addresses the review bottleneck that AI-generated code creates. [6]
Agentic Toolbox lets coding agents use Qodo’s rules and codebase context while they write. This means company standards are applied before a pull request is even opened. The rules come from the organization’s own repositories, so agents work within the right guardrails from the start. [6]
Software Map shows how repositories and services connect. It automatically maps dependencies, service relationships, and contracts. It also calculates the “blast radius” of a proposed change, so teams can see what might break. Quality issues appear as a heat map, making technical debt visible. [6]
Analytics Dashboard shows how the team responds to Qodo findings. It tracks impact (how findings lead to accepted changes), compares activity by repository, and analyzes responses by PR author. This gives managers a clear picture of where quality is improving and where it is not. [6]
Wisdom Base is the knowledge layer underneath everything. It keeps a continuously updated understanding of the codebase, standards, architecture, and pull request history. This is what makes the other features smart. It learns from the organization’s own patterns rather than applying generic rules. [6]
Enterprise deployment options make adoption easier. There is an onboarding wizard, review presets tuned to different goals (catching more issues vs. reducing noise), rules import from specific repositories, and support for air-gapped or on-premises deployment. [6]

Licensing

Qodo uses an open-core model. The core pull request review engine, known as PR-Agent, is open source. It was originally launched under the Apache 2.0 license in 2023. In April 2026, Qodo transferred stewardship of PR-Agent to a community-owned GitHub organization and restored the Apache 2.0 license, moving away from any restrictive terms. This means anyone can use, modify, and distribute the PR-Agent engine freely, and teams can self-host it by supplying their own LLM API keys and compute. [7]

The hosted Qodo Merge platform, along with the enterprise governance features introduced in Qodo 3.0, is closed and proprietary. Pricing follows a per-seat subscription model: a free Developer tier provides 30 PR reviews and 250 IDE/CLI credits per month, the Teams tier costs $30 per active user per month (billed annually), and Enterprise is custom-negotiated with options for self-hosted or on-premises deployment. This hybrid approach gives developers an open-source foundation to build on while providing enterprises with a commercial product that includes governance, analytics, and support. [8]

Why Choose Qodo?

The simple answer is this: Qodo understands that generating code and governing code are different problems. Most tools focus on generation. Qodo focuses on verification, quality, and governance.

Its edge comes from three things: deep context (it understands the whole system, not just one file), enterprise focus (it is built for large organizations with complex codebases), and a team with the right experience (founders who have built and sold companies, advisors from OpenAI and Meta, and investors who understand the space).

AI agents can help us build software faster. But without control and governance, speed can create risk. Qodo 3.0 aims to provide that control. It lets agents work quickly while humans stay in charge. That is the balance modern software teams need.

References

Qodo. “Beyond Intelligence: Qodo’s $70M Series B and the Shift to Artificial Wisdom.” Qodo Blog, March 30, 2026. https://www.qodo.ai/blog/qodo-70m-series-b-shift-to-artificial-wisdom/
Qodo. “We’re on a mission to make code integrity simple.” Qodo About Page. https://www.qodo.ai/about/
Qodo. “Qodo | AI Agents for Code, Review & Workflows.” Qodo Homepage. https://www.qodo.ai/
Qodo. “Critical Capabilities Report.” Qodo Reports, November 21, 2025. https://www.qodo.ai/reports/gartner-critical-capabilities-ai-code-assistance-2025/
Qodo. “What’s new - Qodo Documentation.” Qodo Docs, October 1, 2026. https://docs.qodo.ai/whats-new
Qodo. “Qodo 3.0 Puts Governance at the Center of Agentic Code.” Futurum Group, October 2, 2026. https://futurumgroup.com/qodo-3-0-puts-governance-at-the-center-of-agentic-code/
Qodo. “Qodo Is Handing PR-Agent Over to the Community.” Qodo Blog, April 23, 2026. https://www.qodo.ai/blog/qodo-is-handing-pr-agent-over-to-the-community/
API Evangelist. “Qodo Plans and Pricing.” API Commons, June 21, 2026. https://raw.githubusercontent.com/api-evangelist/qodo/refs/heads/main/plans/qodo-plans-pricing.yml
Qodo. “Qodo Raises $70M in Series B Funding.” FinSMEs, March 30, 2026. https://www.finsmes.com/2026/03/qodo-raises-70m-in-series-b-funding.html
Qodo. “Qodo Ranked #1 AI Code Review Tool in Martian’s Code Review Benchmark.” Qodo Blog, March 15, 2026. https://www.qodo.ai/blog/qodo-ranked-1-ai-code-review-tool-in-martians-code-review-benchmark/

The Sovereignty Illusion: What Companies Get Wrong About Cloud Independence

Haithem Slimi — Fri, 02 Oct 2026 15:06:09 GMT

This is the third chapter of Built to Leave, a series on cloud dependency, sovereignty, and the architecture that makes exit possible. The series follows the problem from its origins — the build-vs-buy trade-off that predates AI — through the structural costs of cloud dependency, the work required to regain control, and the architectural shift that AI workloads are now forcing. Full table of contents: nightthoughts.hashnode.dev/build-to-leave

In the first chapter of this series, I looked at the oldest trade-off in enterprise technology: build it yourself, or buy it from someone else. In the second, I looked at how two forces—defensive security and offensive competitiveness—are pushing financial institutions toward the same infrastructure at the same time. This chapter looks at the foundation both of those forces run on, and the uncomfortable question underneath it: whose cloud is it, really?

There's a phrase that captures the core misunderstanding I want to unpack here: "Your data, their laws." It sounds provocative, maybe even alarmist. But for a European company storing data with a US-headquartered cloud provider, it is a literal description of the legal reality. The US CLOUD Act and FISA allow US authorities to compel US-incorporated companies to hand over data—regardless of where that data is physically stored. A server in Frankfurt does not put that data beyond the reach of US law if the company operating it is subject to US jurisdiction.

That's the sovereignty illusion. For years, "sovereign cloud" has been sold to European companies as a matter of data residency: keep the bits inside EU borders, and you've solved the problem. But jurisdiction follows the company, not the server. And as long as the parent company answers to another government's laws, "complete sovereignty" remains out of reach.

This matters most for Europe, where the gap between ambition and reality is widest. US hyperscalers control over 70% of the European cloud market. European providers hold around 15%, a share that has barely moved despite years of investment and political will. DORA has given European supervisors new powers over critical third-party providers, but oversight is not the same as alternatives. And the concentration risk isn't theoretical—the October 2025 AWS US-East-1 outage took down European banks, brokerages, and payment systems, a preview of what's at stake when the infrastructure you don't control fails.

For US or Chinese companies, this is rarely a live question—they operate under the same jurisdiction as their dominant cloud providers. For European companies, it is unavoidable. So what does "independence" actually mean when both your security posture and your competitive AI capability depend on infrastructure owned by someone else, operating under someone else's laws? That's the question this chapter sits with. It doesn't have a tidy answer—but the first step is naming the problem clearly: your data, their laws. Everything else follows from there.

Why Public Cloud Is Non-Negotiable

Before getting into the risks, it's worth being honest about why European companies can't simply walk away from public cloud. The dependency isn't a failure of strategy; it's a rational response to three converging pressures.

AI requires scale. Generative AI has deepened dependency across three layers: specialized chips (Nvidia-dominated), foundation models (GPT, Gemini, Claude), and integrated AI platforms from hyperscalers. Building that stack in-house is not a realistic option for most companies. The compute requirements alone—training runs costing tens of millions, inference at scale requiring global distribution—put it out of reach for any institution whose core business is finance, not infrastructure.

Resilience demands it. Rising climate and geopolitical risks are pushing infrastructure resilience requirements toward multi-region, geographically diversified setups almost as a baseline expectation now, not a best practice for the ambitious. One global analysis of nearly 9,000 data centers found that a meaningful share already sit in locations facing high or moderate physical climate risk. A separate analysis put the share of capacity exposed to acute climate hazards—flooding, extreme wind, wildfire—at close to four in five facilities worldwide. Meeting that bar in-house means duplicating infrastructure across regions, at a cost and complexity that grows with every added region.

Cost efficiency is real. For years, US hyperscalers offered European companies unmatched scalability and cost efficiency. The pay-as-you-go model converts large fixed investments into variable costs. For institutions under margin pressure, that flexibility is not a luxury—it's a competitive necessity.

Regulatory pressure cuts both ways. DORA requires robust operational resilience, but it also acknowledges that cloud concentration is a systemic risk. The regulation gives supervisors new powers over critical third-party providers, but oversight is not the same as alternatives. A covered entity using a designated provider gets some assurance that the provider is being supervised, but the covered entity remains accountable for its own risk management.

The point is this: public cloud isn't a choice European companies can undo. The question is what that dependency actually costs—and whether the price is being measured honestly.

The Risks: What Companies Are Actually Paying For

The Jurisdiction Trap

The US CLOUD Act and FISA allow US authorities to compel US-incorporated companies to hand over data regardless of where it's stored. This creates a direct conflict with GDPR and the EU's data protection framework. A server in Frankfurt is legally an American server. Data residency only answers where data sits, not who can be compelled to hand it over.

This isn't a theoretical concern. The CLOUD Act was designed precisely to resolve conflicts between US law enforcement demands and foreign data protection laws—and it resolves them in favor of US access. For a European company, that means the legal protections you've built around data residency can be overridden by a legal order you have no standing to contest.

Concentration Risk

According to the Dutch Authority for the Financial Markets (AFM) and De Nederlandsche Bank (DNB), the Dutch financial sector faces increasing systemic risks stemming from its growing reliance on a limited number of non-European IT service providers. The regulators caution that this dependency amplifies the risk of concentration and systemic disruption, where a failure at a single provider could impact large segments of the financial sector (AFM & DNB, 2025).

More than 30% of significant EU institutions' total outsourcing budgets flow to just 10 providers. In 2025, 29% of major ICT incidents reported across the EU financial sector originated from third-party provider failures. The October 2025 AWS US-East-1 outage took down banks, brokerages, and payment systems across the continent—a preview of what happens when infrastructure you don't control fails.

The concentration problem isn't just about outages. It's about the fact that a handful of providers now sit underneath the operational resilience of the entire European financial system. When many institutions lean on the same few providers, a single failure—technical, legal, or geopolitical—ripples across all of them at once.

Geopolitical and Regulatory Exposure

Model availability can be restricted through US sanctions or export controls. Sensitive data and metadata may be exposed to third-country access. Access to cloud services, updates, or AI models could be restricted in a crisis scenario.

This is the risk that's hardest to price because it's the hardest to imagine. But it's not hypothetical. Export controls have already been used to restrict access to advanced chips and AI models. Sanctions can sever access to services overnight. Data localization mandates can force abrupt architectural changes. For a European company that has built its AI strategy on a US hyperscaler's platform, a geopolitical rupture isn't just a business continuity problem—it's an existential one.

The Cost of Sovereignty

According to Eric Bierry, CEO of SBS and Deputy CEO of 74 Software, when CDC, the French public bank, migrated a regulatory reporting service from AWS to France's sovereign SecNumCloud infrastructure, the cost more than tripled—even though the underlying product and service stayed exactly the same (SBS Software, 2026).

True 100% sovereignty isn't achievable anyway. Even domestic infrastructure relies on non-European chips and software somewhere in the stack (SBS Software, 2026). The question isn't whether you can achieve perfect sovereignty—you can't—but whether you're making deliberate trade-offs between cost, control, and risk.

The Compliance Burden

The US CLOUD Act and FISA enable government access to data physically held in the EU where a US nexus exists. This creates a direct conflict with GDPR and the EU's data protection framework. Companies are caught between two legal regimes that make incompatible demands: US law says hand it over; EU law says you can't.

That conflict doesn't resolve itself through better contracts or more careful data mapping. It's structural. And it's the reason "sovereign cloud" keeps coming up in board conversations that would rather be about something else.

What Companies Can Do: Recommendations

The risks are real, but they're not unmanageable. What follows is a set of practical recommendations drawn from regulators, industry bodies, and real-world migrations—not a solution set, but a starting point for institutions that need to act while the strategic questions are still being worked out.

1. Map concentration risk by service, not just provider. According to AFME's 2021 paper on cloud computing, banks need to proactively architect for greater resilience by mapping dependencies between services and geographies—identifying, for example, where two different services share a single point of failure, or how an outage in one region may affect the underlying cloud service provider's control plane (AFME, 2021). Concentration risk is an aggregate of your underlying threat scenarios. You need to evaluate each service dependency individually, not just count how many hyperscalers you use.

2. Adopt a hybrid and multi-cloud strategy. AFME's recommendations emphasize that banks should maintain an inventory of cloud arrangements and take a risk-based approach using a range of approaches, including multi-cloud and portability (AFME, 2021). In practice: sensitive and regulated data stays within EU-based sovereign clouds, critical systems remain on-premises for maximum control, and less sensitive workloads continue on US hyperscale platforms.

3. Build a placement and classification logic. DORA requires financial entities to maintain an ICT third-party risk strategy and policy for ICT services supporting critical or important functions (DORA). Classify applications, data, and processes by criticality, regulatory sensitivity, data confidentiality, substitutability, and dependency risk. This determines where each workload should be deployed. EU-based operations may be appropriate for critical services, but they do not guarantee sovereignty on their own—you must also assess control rights, support models, access paths, encryption concepts, and contractual exit options.

4. Establish and test exit strategies before go-live. DORA Article 28(8) requires supervised entities to develop transition plans enabling them to remove contracted ICT services and data from third-party providers and securely transfer them to alternatives or in-house (DORA). According to the ECB's supervisory guide, exit plans must be realistic, viable, based on plausible scenarios, and include a planned execution timeline compatible with contractual exit clauses (ECB, 2025). Their feasibility should be tested with independent verification.

5. Maintain independent control over encryption keys. According to the European Banking Authority's outsourcing guidelines (EBA/GL/2019/02), financial institutions must retain effective control over encryption keys protecting sensitive data processed or stored by third-party service providers (EBA, 2019). Without direct key control, institutions cannot demonstrate data sovereignty, execute exit strategies, or guarantee recovery during vendor failures or disputes (EBA, 2019). The EBA treats encryption key control as a fundamental prerequisite for maintaining operational resilience and data sovereignty when delegating functions to external service providers.

6. Negotiate collectively. The AFM and DNB have explicitly warned that widespread reliance on the same providers and infrastructures has led to concentration and systemic risks, and they urge institutions to prepare for disruptive scenarios by collaborating with IT vendors, authorities, and peers (AFM & DNB, 2025). A single company negotiating alone with Amazon or Microsoft has "hardly anything to say"—a few hundred together do.

7. Invest in European sovereign providers where it makes strategic sense. The ECB itself chose OVHcloud to provide sovereign cloud services for the digital euro project, with infrastructure operated entirely within the European Union (OVHcloud, 2026). De Nederlandsche Bank (DNB) signed a contract with Stackit, the cloud platform owned by Germany's Schwarz Group, to reduce its dependence on American cloud companies (Techzine, 2026). Smaller institutions have fewer resources but also lower barriers to adopting European providers—and regulators reward early movers, so an apparent weakness can become a strategic strength.

8. Treat digital sovereignty as a board-level concentration risk topic. The AFM and DNB emphasize that reducing digital dependence is a long-term challenge requiring coordinated European solutions, and they call for the development of robust European alternatives to non-EU IT providers (AFM & DNB, 2025). It connects operational resilience, outsourcing governance, technology strategy, data protection, AI adoption, and geopolitical risk management. The question is no longer whether companies should use global technology platforms, but whether they can use them without losing transparency, decision rights, portability, and credible exit options for their most critical services.

The Question That Doesn't Go Away

None of these recommendations solve the underlying problem. They manage it. They buy time. They reduce exposure at the margins. But they don't change the fundamental reality: European companies depend on infrastructure owned by companies that answer to another government's laws.

That dependency isn't going away. Public cloud is too central to AI competitiveness, too embedded in resilience planning, too cost-effective to abandon. The question isn't whether to use it—it's whether the terms of that use can be renegotiated in a way that gives European companies meaningful control over their own security, their own data, and their own competitive future.

The next chapter looks at what solutions actually exist—not just risk management, but structural alternatives. What would a genuinely sovereign cloud stack look like? What's being built, what's working, and what's still missing? And what would it take for European companies to have real options, not just better contracts?

That's where we go next.

References

PIFS International, "Cloud Adoption in the Financial Sector and Concentration Risk"
Office of the Superintendent of Financial Institutions (OSFI), "Third-Party Risk Management Guideline"
Bristows, "AWS US-EAST-1 incident: regulators concentrate on concentration risk"
calQrisk, "Outsourcing and Third-Party Risk Management for Financial Firms"
FINRA, "Third-Party Risk Landscape," 2025 Annual Regulatory Oversight Report
Data Center Dynamics, "Climate threats to data centers set to surge"
XDI, "Global data centres face rising climate risks"
European Supervisory Authorities, DORA CTPP Designation List (November 2025) — https://www.eba.europa.eu/publications-and-media/press-releases/european-supervisory-authorities-designate-critical-ict-third-party-providers-under-digital
ECB, Supervisory Guide on Cloud Outsourcing (July 2025) — https://www.bankingsupervision.europa.eu/ecb/pub/pdf/ssm.supervisory_guides202507.pt.pdf
EBA, Outsourcing Guidelines (EBA/GL/2019/02) — https://www.kiteworks.com/regulatory-compliance/eba-encryption-key-control-guidelines/
AFM & DNB, "AFM and DNB warn of systemic risks in the financial sector from digital dependence" (October 2025) — https://www.afm.nl/en/sector/actueel/2025/okt/pb-digitale-autonomie
OVHcloud, "OVHcloud to provide sovereign cloud services for the ECB digital euro" (March 2026) — https://corporate.ovhcloud.com/en-ca/newsroom/news/ovhcloud-digital-euro-ecb/
Techzine, "Dutch central bank chooses Lidl for European Cloud" (April 2026) — https://www.techzine.eu/news/infrastructure/140634/dutch-central-bank-chooses-lidl-for-european-cloud/
SBS Software, "AI in banking | Part 3: Why '100% sovereign' is a myth" (September 2026) — https://sbs-software.com/insights/artificial-intelligence-data/podcast-ai-in-banking-why-100-sovereign-is-a-myth/
AFME, "Cloud Computing in Capital Markets" (June 2021) — https://www.afme.eu/Portals/0/DispatchFeaturedImages/AFME_CloudComputing2021_06-2.pdf
FISA (Foreign Intelligence Surveillance Act), including Section 702 provisions — https://www.law.cornell.edu/wex/foreign_intelligence_surveillance_act
US CLOUD Act — https://www.congress.gov/bill/115th-congress/senate-bill/2383

Two Races at Once: Why AI Is Both a Security Emergency and a Competitive One

Haithem Slimi — Fri, 02 Oct 2026 13:43:56 GMT

This is the second chapter of Built to Leave, a series on cloud dependency, sovereignty, and the architecture that makes exit possible. The series follows the problem from its origins — the build-vs-buy trade-off that predates AI — through the structural costs of cloud dependency, the work required to regain control, and the architectural shift that AI workloads are now forcing. Full table of contents: nightthoughts.hashnode.dev/build-to-leave

The previous piece ended with two forces pulling financial institutions toward the same infrastructure, for very different reasons: one defensive, one competitive. This piece looks at what that means once both land on the same roadmap at the same time, based on what I've seen in cloud transformation work, kept at the general level rather than tied to any one employer.

Security is not one bank's problem

Security in a bank is never negotiable. A serious breach doesn't stay contained — banks are wired into each other, and a failure at one institution can ripple into all of them. That's why the sector's response is increasingly collective, not institution by institution. Project Glasswing, launched by Anthropic in April 2026, brings major banks and hyperscalers together — JPMorgan, AWS, Google, Microsoft, Palo Alto Networks — to jointly use AI to find and fix vulnerabilities before attackers do, and by June 2026 it had expanded to roughly 150 more partners across 15+ countries.

The urgency behind that effort just became a lot more concrete. On September 3, 2026, OpenAI released GPT-6 Astra — the first commercial AI model to reach OpenAI's own "Critical" threshold for cybersecurity capability. According to OpenAI's system card, Astra can find previously unknown security flaws and develop new ways to exploit them across well-protected systems, without a person guiding each step. In pre-release testing it discovered two zero-day vulnerabilities on its own. This is no longer hypothetical. As of this month, that capability exists commercially, under gated access. Fighting it requires AI-capable defense, alongside the discipline that comes with it: zero trust as a default, continuous scanning across binaries, source code, third-party libraries, and internal systems, and relentless tracking of vulnerabilities before they can be weaponized.

The other pressure

Security isn't the only force reshaping the roadmap. A second pressure is building in parallel, and it decides who's still standing in five years: competitiveness. There are two different things "AI" can mean here. One is tooling — copilots inside email, code, presentations — which makes existing work faster without changing what the business does. The other is AI as a business function: pricing, credit scoring, fraud detection — AI that changes the product itself. That second category is what actually moves the competitive needle.

Institutions that adopt it early are already pulling ahead on cost and performance; slower movers risk a cost base they can't compete with. Multiple surveys show banks now ranking cost reduction and productivity ahead of security as reasons to invest in AI. Even inside institutions that treat security as non-negotiable, the case for AI is increasingly built around staying competitive, not just staying safe.

Where I've seen this actually play out

The technology was rarely the hardest part. Two things running in parallel were.

The first is applying real security discipline against a technology landscape that, in most institutions I've seen, was never built with that discipline in mind. You can't zero-trust your way through hundreds of small, loosely documented applications overnight, especially against an adversary that can now automate vulnerability discovery at machine speed. Every one of those applications is its own scanning target, its own audit trail, its own zero-day exposure to track. Multiply a hard problem by a fragmented estate, and it becomes close to unmanageable regardless of budget.

The second is organizational: getting an institution to agree, in practice and not just in principle, on what gets funded first when security and AI competitiveness are both labeled the top priority. They compete for the same architects, the same budget cycle, the same production windows. That tension doesn't resolve through better slogans. It resolves by removing the reason it exists.

The fix nobody wants to do first

Most institutions aren't failing to prioritize security or competitiveness. They're trying to do both against an application landscape that should never have gotten this fragmented. Every small, tactically-built application — built to hit a deadline, wrapped around

The Old Trade-Off: Vendor vs. In-House, Before AI Changed the Rules

Haithem Slimi — Fri, 02 Oct 2026 13:37:43 GMT

This is the first chapter of Built to Leave, a series on cloud dependency, sovereignty, and the architecture that makes exit possible. The series follows the problem from its origins — the build-vs-buy trade-off that predates AI — through the structural costs of cloud dependency, the work required to regain control, and the architectural shift that AI workloads are now forcing. Full table of contents: nightthoughts.hashnode.dev/build-to-leave

Long before anyone in finance was talking about AI, institutions were already wrestling with one of the oldest questions in enterprise technology: build it yourself, or buy it from someone else. It's a decision every bank, insurer, and financial services firm has made repeatedly, across decades, for every new capability that came along — core banking systems, trading platforms, risk engines, data infrastructure. The technology changed. The underlying dilemma didn't.

This article isn't about AI yet. It's about understanding the trade-off as it existed before AI entered the picture — because you can't understand how AI is reshaping the game until you understand the board it's being played on.

A brief chronology

Financial institutions have swung between these two poles for a long time, and rarely by pure technical logic — often by cost pressure, regulation, or the scars of a previous decision gone wrong.

In earlier decades, many institutions built almost everything in-house, out of necessity more than preference — the vendor ecosystem for specialized financial technology simply didn't exist yet. Core systems were custom, maintained by large internal IT departments, because there was no market offering an alternative.

As the vendor landscape matured, the pendulum swung toward buying. Specialized providers emerged for almost every function — trading systems, compliance tooling, payment processing, data feeds — and outsourcing that work became the default, because building it yourself no longer made economic sense against a mature, specialized alternative.

More recently, the pendulum has been swinging back, at least partially. Institutions that outsourced heavily started reckoning with the risks of that dependency, and many began reinvesting in in-house engineering capability — not to replace vendors entirely, but to regain control over what they saw as strategically important.

Neither swing was permanent, and neither was universally correct. What's stayed constant is the underlying trade-off itself.

The case for the vendor

Buying from an external provider has always come with a clear pitch: you get expertise and speed without having to build the capability yourself.

Cost predictability (at least on paper). You pay for a service rather than building and maintaining the underlying capability, converting a large fixed investment into a more variable one.
Specialized expertise. Vendors often have deep, focused expertise in a narrow domain that would take years for an internal team to replicate.
Faster time to market. A mature vendor product can often be deployed faster than a custom-built equivalent, because much of the engineering work is already done.
Shared risk, in theory. SLAs and contracts create an expectation that the vendor shares responsibility for uptime, security, and support.

But this pitch has always come with a quieter set of liabilities, ones that only become visible over time:

You don't control the roadmap. The vendor's priorities are not necessarily your priorities, and you may find yourself waiting on features, fixes, or changes that matter to you but not to their broader customer base.
Dependency risk compounds. The longer you rely on a vendor, the more expensive and disruptive it becomes to leave — even if the relationship deteriorates.
The vendor can fail you in ways outside your control. Contract terms can change unilaterally once switching costs are high enough. Vendors can be acquired, restructured, or go bankrupt, taking your roadmap down with them. A security breach or outage on their side becomes an incident on yours, regardless of whose code was vulnerable — regulators such as FINRA have specifically flagged the rise in cyberattacks and outages at third-party vendors as a risk that can ripple across large numbers of firms at once (FINRA). And increasingly, geopolitical shifts — sanctions, export controls, data residency requirements — can sever access to a vendor's service almost overnight, regardless of what the contract says.
You become the auditor, not the owner. Choosing a vendor doesn't remove your responsibility — regulators make that explicit in financial services. Outsourcing a function does not outsource the accountability for it: if a critical provider fails, the institution's own customers feel the consequences and its own regulator expects a response, including formal incident reporting where thresholds are met (calQrisk). The EU's Digital Operational Resilience Act (DORA), in force since January 2025, codifies exactly this: it gives supervisors the power to designate systemically important ICT and cloud providers as "critical third-party providers" subject to direct oversight, precisely because of the concentration risk created when many institutions lean on the same few providers (PIFS International; Bristows). Canada's OSFI guidance is just as direct, advising regulated institutions to consider strategies such as multi-cloud design specifically to mitigate cloud-provider concentration risk (OSFI). That's the real substance behind "you become the auditor" — ongoing due diligence, SLA monitoring, security assessments, and resisting the temptation to let any single vendor become a single point of failure.

The case for in-house

Building internally has always promised the opposite: control, at a price.

Full ownership of the stack. No dependency on a third party's roadmap, pricing decisions, or continued existence.
Institutional knowledge stays inside the company, compounding over time rather than walking out the door when a vendor contract ends.
Tighter alignment with regulatory and compliance requirements, since governance sits entirely within the institution's own control rather than being negotiated through a vendor relationship.
Faster internal feedback loops, at least once a team is mature — the people who build a system are also the ones maintaining and evolving it, without a vendor relationship as an intermediary.

And the costs, while less visible than a vendor invoice, are just as real:

Fixed cost that doesn't scale down. Infrastructure and talent are ongoing commitments, whether the system is running at 20% capacity or 90%.
Slower delivery, particularly in regulated environments. Every internal change typically passes through governance, audit trail requirements, and compliance review — overhead that a mature vendor product may have already absorbed into its design and certifications.
Mission drift. For a financial institution, IT is not the core business. The more capability is brought in-house, the more the institution is, in effect, also becoming a software organization — competing for technical talent, carrying technical debt, and needing a level of engineering maturity that doesn't come naturally to a business built around finance, not software.
Self-auditing is still auditing — and it's easier to under-invest in. Owning the stack doesn't remove the need for rigorous internal controls; it just moves that function inside the institution's own walls, where there's no external contract forcing the discipline. In-house risk tends to fail quietly, through neglect, while vendor risk tends to fail loudly, through breach disclosures and contract disputes — and that asymmetry in visibility often shapes the debate more than the actual underlying risk does.
The operational headache no slide deck shows. Owning your IT end-to-end means owning problems that have nothing to do with writing good software. Running global operations means organizing a genuine follow-the-sun strategy — coverage across time zones, handovers that don't lose context, incident response that works at 3 a.m. somewhere no matter what. It means spreading knowledge across a distributed workforce rather than concentrating it in a single team, which is harder to do well than it sounds. And paradoxically, the more IT you own, the closer IT has to sit to the business — which means growing the very "bigger IT" organization that the mission-drift argument warns against in the first place.
Data center provisioning is its own discipline. Standing up and running physical infrastructure — capacity planning, power, cooling, hardware refresh cycles, physical security — is a specialized competency in itself, largely unrelated to financial services. Institutions that own this end-to-end are committing real, ongoing effort to keep SLAs at the level the business requires, effort that has nothing to do with banking or finance and everything to do with running a data center well.
Hosting costs that only go one direction. The cost of maintaining and upgrading data center infrastructure tends to climb over time, not fall. Construction costs per megawatt of IT load have risen sharply as demand for power-hungry, high-density facilities has surged, with electrical infrastructure now consuming close to half of total project budgets (Terrapin CG; iRecruit). Power itself has become the binding constraint: grid operators and energy researchers point to data center demand as a meaningful driver of rising electricity costs in several markets (Fortune). Running infrastructure at the resilience level financial regulators expect isn't cheap, and the trend line isn't moving in the direction that makes self-hosting easier over time.
Resiliency requirements are escalating faster than most in-house setups can absorb. Geopolitical instability and worsening climate conditions — more frequent flooding, wildfires, earthquakes, and extreme weather events — are pushing infrastructure resilience requirements toward multi-region, geographically diversified setups almost as a baseline expectation now, not a best practice for the ambitious. This isn't a theoretical concern: one global analysis of nearly 9,000 data centers found that a meaningful share already sit in locations facing high or moderate physical climate risk, with several major hubs projected to see a large proportion of their facilities exposed by 2050 (DCD; XDI). A separate analysis of global data center markets put the share of capacity exposed to acute climate hazards — flooding, extreme wind, wildfire — at close to four in five facilities worldwide (CNBC). Meeting a rising resilience bar in-house means duplicating infrastructure across regions, at a cost and complexity that grows with every added region. At a certain point, this pressure alone pushes institutions to conclude that running physical infrastructure at that level of resiliency is better delegated to an organization for whom that is the core business — which is precisely the case public cloud providers make.

The speed trap: how both paths lead to the same mess

There's a dimension to this trade-off that doesn't show up in cost comparisons or risk matrices, but that anyone who has actually shipped software in a bank will recognize immediately: speed pressure distorts both paths, and it tends to distort them toward the same outcome.

When the business sees an opportunity, the instinct is to move before competitors do. On the in-house side, that pressure produces what I'd call commando mode — or sniper mode: build exactly what's needed, nothing more, and move fast. No time for the full picture. No time to future-proof. Just enough to seize the opportunity, now, while it's there. And for day one, this works. It's often the right call — a mature, fully governed build process would have missed the window entirely.

The problem is what happens next. As soon as that commando-mode build reaches production and the business succeeds, the very success creates pressure the system was never built to absorb. It doesn't scale. Clients want more than the minimal version can deliver. The business struggles to understand why something that clearly works can't simply be extended — surely if it does this, it can do that too? And underneath, the parts that were skipped in the rush — proper security, authentication, resilience — are exactly the parts that are hardest and most expensive to retrofit once real usage and real data are flowing through the system.

The vendor path isn't immune to its own version of this trap, just from a different direction. Getting real value out of a vendor platform — configuring it properly, building the features specific to your business — requires expertise, and that expertise is often scarce and expensive. Institutions frequently end up dependent on a small number of specialists who understand the vendor's tool deeply, and just as often end up building a growing constellation of satellite applications around the vendor platform to bridge the gaps it doesn't cover natively. Ironically, by the time all of that is stitched together, time to market on the vendor path can end up slower than a commando-mode in-house build — the very speed advantage vendors are supposed to offer gets eaten by integration and customization overhead.

Here's the uncomfortable convergence: whichever path you take, the default outcome, without deliberate intervention, is the same — a sprawling landscape of small applications in production, built under time pressure, neither scalable nor properly secured, each one requiring real ongoing effort, spread across teams worldwide, just to keep it up and running.

This is where I think the real discipline lives — not in choosing build or buy correctly in the abstract, but in resisting the urge to let either path run on pure speed indefinitely. At some point, an institution has to deliberately slow down: organize what's been built, structure it properly, size the actual opportunity rather than the emergency that created it, and identify which of these applications are mature and important enough to deserve real investment — and which should be retired, consolidated, or never should have reached production in their current form in the first place.

Neither side is "safe"

What's striking, looking back across this history, is how often the debate gets framed as if one side is the "risky" choice and the other is the "safe" one. It never has been. Buying and building simply distribute risk differently — one concentrates it in a third-party relationship you have to actively manage, the other concentrates it inside your own walls, where it's your discipline alone keeping it in check.

The institutions that navigated this well weren't the ones that picked a permanent side. They were the ones that got specific: deciding deliberately which capabilities were core enough to own directly regardless of cost, and which were commodity enough that well-managed vendor risk was the better bet — and revisiting that decision as circumstances changed, rather than defaulting to "how we've always done it."

That balance held, imperfectly but functionally, for decades. But something is changing the terrain underneath it — and not gradually, the way most technology shifts do.

Part of the pressure is infrastructural: resiliency expectations are rising faster than most institutions can build for on their own, quietly pushing even committed in-house organizations toward providers whose entire business is running resilient infrastructure at global scale. Part of it is a discipline problem that predates any of this — the speed trap that fills production with small, fragile applications regardless of which path built them. And part of it is genuinely new, arriving from two directions at once.

One is defensive: the attacker on the other side of the wall is no longer just a person trying a few things and giving up when they fail. The other is competitive, and arguably the more urgent of the two: institutions that learn to use AI well are already starting to produce things faster, at lower cost, with better products, than those that don't. That's not a security question at all — it's a market-share question. An institution can have flawless security and still lose ground simply by being slower and more expensive than a competitor who figured out how to fold AI into their delivery cycle.

Both pressures point toward the same infrastructure — the data, compute, and platforms that make AI viable at scale — but for very different reasons. One is about not losing what you have. The other is about not falling behind. A few open questions to sit with before the next piece:

If resiliency and infrastructure ownership are already pushing institutions toward the cloud on their own, what happens when both defending against an always-on attacker and keeping up with AI-native competitors make that pull even harder to resist?
Is the "slow down and get organized" discipline this article calls for even compatible with a competitive environment where the institutions moving fastest with AI are also the ones pulling ahead?
And if the answer increasingly runs through public cloud — whose infrastructure is it, really, and what does that mean for an institution's independence when both its security and its competitiveness depend on it?

What happens to this trade-off when infrastructure itself becomes too demanding to own alone, discipline becomes harder to enforce under competitive pressure, the attacker never sleeps, and standing still means falling behind? That's where we go next.

References

PIFS International, "Cloud Adoption in the Financial Sector and Concentration Risk"
Office of the Superintendent of Financial Institutions (OSFI), "Third-Party Risk Management Guideline"
Bristows, "AWS US-EAST-1 incident: regulators concentrate on concentration risk"
calQrisk, "Outsourcing and Third-Party Risk Management for Financial Firms"
FINRA, "Third-Party Risk Landscape," 2025 Annual Regulatory Oversight Report
Data Center Dynamics, "Climate threats to data centers set to surge"
XDI, "Global data centres face rising climate risks"
CNBC, "Data center climate change study"
Terrapin CG, "Average Cost to Build a Data Center in the USA"
iRecruit, "Data Center Construction Cost Trends 2026"
Fortune, "Data centers and electricity costs"

The Secret Digital Workers Running the Internet: A Guide to Non-Human Identities

Haithem Slimi — Tue, 29 Sep 2026 22:14:41 GMT

Imagine walking into a modern digital hotel. You use a plastic key card to unlock your room door. That key card represents your human identity—it proves who you are and gives you permission to enter.

However, behind the scenes, hundreds of automated systems are moving without human intervention: luggage delivery bots unlock elevator doors, security cameras check authority passes, and automated payment kiosks talk to bank servers.

To open those doors, these machines need their own keys.

In the digital world, humans are no longer the only ones logging in. Computer programs, automated scripts, software tools, and AI agents log in millions of times per second. These machine passes are called Non-Human Identities (NHIs).

In fact, in modern enterprise environments, non-human identities outnumber human identities by a ratio ranging between 45:1 and 140:1 1.

What Exactly is a Non-Human Identity (NHI)?

At its simplest, a Non-Human Identity is a set of digital credentials—such as an API key, a service account, or a digital token—that allows one software program to speak to another program without a human having to type in a password 2.

An API Key: A special digital password that lets your weather phone app fetch live updates from a weather server.
A Service Account: A background login created so a cloud service can automatically back up your files at midnight.
A Digital Certificate: An encrypted pass that proves a website or device is legitimate and safe to connect with.

Without NHIs, the modern internet would grind to a halt because humans would have to manually approve every single background data exchange.

The 4 Places Where Unmanaged Machine Keys Hide

Because these machine keys are created automatically, they are easy to forget. Security teams face four major challenges when trying to manage them:

Cloud-Native Growth: As companies move to the cloud, billions of tiny software connections are made across multiple servers 2.
Software Assembly Lines (CI/CD Pipelines): Developers use automated tools to build and test code continuously. At every step of this automated assembly line, software tools leave behind digital access keys 1.
Smart Devices & Forgotten Systems (IoT): Connected hardware, old dormant accounts, and expired digital certificates sit silently in networks, still holding active master access 2.
Supply Chain Connections: Modern apps rely on third-party vendor tools. If an app trusts a vendor's API, it creates inherited trust. If that vendor gets hacked, the attacker can walk right into your system through that trusted connection 1.

The Emerging Threat: Logging In with "Borrowed Trust"

Security leaders have noticed a major shift: 80% of security leaders rank AI and machine-related identity risks as their top concern 3.

Why? Because tricking a human into giving up a password takes effort, and humans often have two-factor authentication (like a text message code).

Machine keys, however, rarely have two-factor checks. If a hacker finds a forgotten API key accidentally published in code or stored on a server, they don't need to break in. They simply "log in" using borrowed trust—the system assumes the attacker is just another friendly internal software tool 2.

The Agentic AI Escalation

The risk becomes even bigger with the rise of Agentic AI.

Traditional software only follows rigid, step-by-step instructions. Agentic AI, by contrast, makes autonomous decisions to complete complex goals 2:

It can execute financial transactions.
It can manage supply chain inventory and place orders automatically.
It can interact directly with industrial control systems.

The Danger: Context Loss in Multi-Agent Chains

When AI Agent A hires AI Agent B, which then triggers AI Agent C to perform a financial trade, software systems can experience context loss. They lose track of who originally authorized the command, making it easy for mistakes or malicious instructions to slip through unnoticed 2.

The WEF Safety Framework for Machine Identities

To help organizations protect themselves, cybersecurity experts at the World Economic Forum (WEF) and its Global Future Councils designed a five-step governance framework specifically for non-human identities and agentic AI 2:

1. Universal Discovery ➔ 2. Eliminate Static Keys ➔ 3. Extend Zero Trust ➔ 4. Behavior Anomaly Detection ➔ 5. Identity Delegation Tracing

1. Universal Discovery & Ownership

You cannot protect what you cannot see. The WEF framework emphasizes that organizations must inventory every single API key, bot, and AI agent, and assign a responsible human owner to every machine identity 2.

2. Eliminate Long-Lived Secrets

Static passwords that last for years are dangerous. Organizations must phase out static keys and replace them with ephemeral (short-lived) tokens that expire automatically after a few minutes or hours 2.

3. Extend "Zero Trust" to Machines

Never assume a program is safe just because it is inside your network 2.

Continuous Authorization: Continuously re-verify machine identities.
Strict Least-Privilege Scoping: Give a machine key access only to the specific folder or task it needs—nothing more.
Post-Quantum Cryptography: Upgrade security standards so future quantum computers won't be able to crack machine keys 2.

4. Behavior-Based Anomaly Detection

Instead of just checking if a key is valid when it logs in, systems must watch how the machine behaves. If a bot that normally checks weather updates suddenly attempts to download a customer database at 3 AM, the system must immediately flag and block it 2.

5. Identity Delegation Tracing

As AI agents pass commands down a chain of other tools, temporary safety passes must be attached at every step. This creates a full audit trail so humans can always review exactly why an AI agent made a specific decision 2.

Summary

As AI shifts from simple assistants to autonomous agents that take action on our behalf, protecting machine keys is no longer just a technical detail—it is the foundation of digital safety. Frameworks like the World Economic Forum's give us a roadmap to manage these digital workers safely as our systems grow.

References

[1] Enterprise NHI Statistics (45:1 to 140:1 Ratio): Cloud Security Alliance (CSA) & Entro Security Research. Data shows non-human identities outnumber human users by 45:1 in average enterprise setups and over 140:1 in cloud-native environments.

[2] World Economic Forum (WEF) NHI Framework: World Economic Forum Centre for Cybersecurity. Research and governance guidelines published by the Global Future Councils, titled "Governing Non-Human Identities & Agentic AI."

[3] CISO Threat Landscape Surveys: Gartner Cybersecurity & CISO Insights. Global Chief Information Security Officer (CISO) Industry Reports on Identity & Access Management (IAM), showing ~80% of security executives rank non-human identity exposure and autonomous AI access as top priorities.

Traditional Vector RAG vs. PageIndex: What's the Difference?

Haithem Slimi — Tue, 29 Sep 2026 21:26:21 GMT

Imagine you need to answer a specific question using a 200-page financial report.

How would you do it? You would open the document, look at the Table of Contents, find the right chapter, turn to those pages, and read the entire section to understand the full context.

Most standard AI search systems do not read documents this way. Instead, they use a process called Retrieval-Augmented Generation (RAG), which cuts documents into tiny, disconnected pieces before searching them.

An open-source framework called PageIndex changes this approach. Here is a clear breakdown of how traditional Vector RAG works, how PageIndex offers a vectorless alternative, and where each approach shines.

What is PageIndex and Who Created It?

PageIndex is an open-source framework designed to give AI models a better way to search and analyze long documents without using vector databases or fixed text chunking. It is known as a vectorless, reasoning-based RAG engine.

PageIndex was created by the team at Vectify AI (led by researchers including Mingtian Zhang and Yu Tang) and released on GitHub.

Instead of converting text into abstract mathematical numbers (vectors), PageIndex converts documents into a hierarchical tree structure—essentially a detailed, machine-readable Table of Contents complete with page boundaries and section summaries.

Traditional Vector RAG vs. PageIndex: Step-by-Step

To understand how PageIndex changes document retrieval, compare how both systems process a complex document.

Process Step	Traditional Vector RAG	PageIndex (Vectorless RAG)
1. Document Ingestion	Cuts documents into small, fixed-size chunks (e.g., 512 words).	Extracts the document's natural structure (chapters, headers, subsections).
2. Information Indexing	Converts text chunks into mathematical numbers stored in a Vector DB.	Builds a hierarchical Tree Index with node summaries and exact page/line coordinates.
3. Query Retrieval	Compares query vectors against chunk vectors using mathematical distance.	An AI model reasons through the tree branch-by-branch to locate the target section.
4. Final Context Delivered	Pulls 1–2 isolated text snippets without surrounding context.	Pulls complete contiguous sections/pages with exact page citations.

Key Benefits of PageIndex

Preserves Full Context: Traditional RAG often cuts a sentence or table in half across chunk boundaries. PageIndex retrieves contiguous physical pages, ensuring the AI model sees the complete thought.
High Traceability: Every answer points back to an exact section and page range (e.g., "Pages 12–14"), making it straightforward to double-check source facts.
No Vector Database Infrastructure: You do not need to set up, maintain, or manage external vector databases like Pinecone or ChromaDB.
Higher Accuracy on Complex Docs: On structured reports, reasoning through section hierarchies yields significantly higher answer accuracy than matching isolated keywords or phrases.

Where PageIndex Isn't Perfect: The Limitations

While PageIndex fixes major flaws in traditional chunk-based retrieval, it is not a solution for every problem. It comes with clear trade-offs:

Higher Latency (Slower Retrieval): Because PageIndex relies on an LLM to read through tree nodes and navigate branches step-by-step, queries take longer than fast vector math lookups.
Higher API Token Costs: Traversing node summaries with an LLM sends more tokens back and forth, which increases the execution cost per query.
Requires Structured Documents: PageIndex relies on documents having a logical layout (headers, titles, clear sections). It struggles with completely unstructured text, poor OCR scans, or unorganized text files.
Not Ideal for Massive Multi-Document Pools: If you need to search across 100,000 separate 1-page customer support emails simultaneously, traditional vector search remains much faster and more scalable.

Choosing the Right Tool for Your Project

Selecting between Traditional Vector RAG and PageIndex depends on your data structure and query requirements.

Use PageIndex when working with long, structured documents—such as financial 10-K filings, legal contracts, technical manuals, or research papers—where accuracy, full context, and exact page citations are required.

Use Traditional Vector RAG when you need sub-second search speeds across thousands of short, unstructured files, or where low latency and lower token costs outweigh the need for deep contextual reasoning.

To explore the codebase and run a quickstart implementation, visit the official repository at VectifyAI/PageIndex on GitHub.

How to Build an Intelligent Router for NVIDIA and Ollama with LiteLLM, JEV, and Memori on a Mac Mini (Part 2)

Haithem Slimi — Mon, 28 Sep 2026 22:57:40 GMT

Part 1 covered the foundation: an NVIDIA API key, a local proxy for NVIDIA NIM, and OpenClaw configured with its native NVIDIA provider. That setup gave the Mac mini access to frontier NVIDIA models, but it kept two separate paths. Local Ollama models lived on localhost:11434. NVIDIA models went through the proxy on localhost:5002. Clients had to know which backend to call.

Part 2 removes that choice. A single intelligent router sits in front of both Ollama and NVIDIA NIM, decides which model serves each request, and adds persistent memory across sessions.

If the foundation from Part 1 is not in place, it is covered here: How to Set Up NVIDIA API Keys and Use Them with Ollama and OpenClaw on a Mac Mini (Part 1).

A Note on the Proxy from Part 1

Part 1 used ollama-proxy to bridge Ollama and NVIDIA NIM. That bridge is no longer needed. LiteLLM handles the same job natively, along with provider abstraction, fallbacks, retries, and routing logic. The proxy becomes redundant.

Retire it once clients point to LiteLLM:

launchctl unload ~/Library/LaunchAgents/com.ollama.proxy.plist

If any tools still call port 5002 with nvidia-nim: prefixed models, keep the proxy running until those clients are updated. Then remove it. Running both long-term adds two configs to maintain and two places for the API key.

Part 1 was a valid stepping stone. Part 2 is the upgrade.

The Stack at a Glance

Three components work together to route between local Ollama models and NVIDIA NIM:

Component	Role
LiteLLM	OpenAI-compatible proxy that routes to 100+ providers, including Ollama and NVIDIA NIM. Replaces `ollama-proxy` from Part 1.
JEV	TypeSafe's System One model. A lightweight decision layer that picks which model serves a request.
Memori	SQL-native memory engine that gives the router persistent, queryable memory across sessions.

Each one is replaceable. LiteLLM handles transport and provider abstraction. JEV handles routing intelligence. Memori handles context and recall.

Architecture

flowchart TB
    A[Clients] --> D[LiteLLM Router]
    D --> E[JEV decision]
    E --> F[Model pool]
    F --> H[Local Ollama]
    F --> I[NVIDIA NIM]
    D --> G[Memori]
    G --> D

The architecture consists of three layers:

Clients send requests to the LiteLLM router on port 4000. Clients include OpenClaw, Ollama clients, and any OpenAI-compatible tool.
The Router is powered by LiteLLM. It uses a pre-call hook to build a request summary. The JEV decision layer picks the cheapest eligible tier. The model pool routes the request to either local Ollama or NVIDIA NIM.
Memory is provided by Memori. It connects to the pre-call hook. It recalls relevant memories before the call and stores the conversation after the call in a local SQLite database.

The router listens on http://localhost:4000. Local Ollama runs on localhost:11434. NVIDIA NIM is reached at https://integrate.api.nvidia.com/v1.

How a Request Flows

Client sends a request to the LiteLLM proxy with the model set to jev-router.
The pre-call hook intercepts the request and builds a minimized summary: recent message roles, truncated text, and signals for images or tool use.
The decision layer picks which tier should serve the request. It returns a tier choice with confidence and probabilities.
LiteLLM routes the request to the selected model in the pool. Local models go to Ollama. NVIDIA models go to NVIDIA NIM.
Memori injects relevant memories before the call and stores the conversation afterward, using a standard SQL database as the backing store.

Why This Architecture

Single Endpoint

Clients no longer need to know which backend to use. They call http://localhost:4000/v1/chat/completions with model: "jev-router". LiteLLM and the decision layer handle the rest, routing to either Ollama or NVIDIA NIM.

One Component Instead of Two

Part 1 required a separate proxy for NVIDIA and direct calls to Ollama for local models. LiteLLM absorbs both. The NVIDIA API key lives in one place. The Ollama endpoint lives in one place. Clients see one URL.

Intelligent Routing

Prefix-based routing is simple but rigid. nvidia-nim: always goes to NVIDIA. llama3.2 always goes to Ollama. A decision layer adds judgment. It looks at the request and picks the cheapest tier whose models can fully answer it.

Persistent Memory

Memori gives the router a memory that survives restarts. It works with any LLM framework and uses standard SQL databases instead of vector stores, which cuts memory costs by an estimated 80-90%.

Security Principles

Before installing anything, three decisions shape the security posture of the entire stack.

Pin LiteLLM to a safe version. LiteLLM had critical vulnerabilities in 2026, including a supply chain incident in March and several CVEs patched in v1.83.0 and later. The version floor for any proxy is >= 1.83.7. Version 1.83.7 fixes CVE-2026-42271, an arbitrary command execution vulnerability. Install with an exact pinned version and verify.

Generate a strong master key. The LiteLLM master key is the proxy's admin credential. Without it, anyone who can reach port 4000 can use every model in the pool. The key must start with sk- and be generated from at least 32 random bytes. Never use the default sk-1234 placeholder. Exposed gateways are a known problem.

Bind to localhost only. LiteLLM binds to 0.0.0.0 by default, which exposes the proxy to the local network. Add --host 127.0.0.1 to every run command and launchd plist.

Use virtual keys for clients. The master key is the admin credential. It must never be handed to a consumer. Clients should use scoped virtual keys, so access can be revoked without rotating the master key.

Disable environment credential login for the Admin UI. If the Admin UI is not needed, do not expose it. If it is needed, set disable_env_credential_login: true in config.yaml after creating a proper admin account.

These principles are applied step-by-step in the sections that follow.

Before You Start: Verify NVIDIA API Key

The NVIDIA_NIM_API_KEYS variable is set in Part 1. If it is already in your shell profile, skip to the next section. If not, or if you are starting fresh, verify it now.

Check the current shell:

echo "NVIDIA_NIM_API_KEYS=${NVIDIA_NIM_API_KEYS:-unset}"

Expected output if set:

NVIDIA_NIM_API_KEYS=nvapi-xxxxxxxxxxxxxxxxxxxx

If it prints unset, add it to your shell profile. For Zsh:

echo 'export NVIDIA_NIM_API_KEYS="nvapi-xxxxxxxxxxxxxxxxxxxx"' >> ~/.zshrc
source ~/.zshrc

For Bash:

echo 'export NVIDIA_NIM_API_KEYS="nvapi-xxxxxxxxxxxxxxxxxxxx"' >> ~/.bash_profile
source ~/.bash_profile

Replace the placeholder with the actual key from build.nvidia.com. The proxy and LiteLLM both read this variable.

Installing the Stack

Prerequisites

The Mac mini already has Python 3.11 or later from Part 1. Verify:

python3 --version

Expected output:

Python 3.11.x

Install LiteLLM

Install the pinned version with proxy support:

pip3 install 'litellm[proxy]==1.83.7'

Verify:

litellm --version

Expected output:

litellm, version 1.83.7

Install Memori

pip3 install memorisdk

Create the memory directory:

mkdir -p ~/ai-router/memory

Install the JEV Router

pip3 install jev-router

Alternatively, clone the repository:

git clone https://github.com/prismhq/jev-router.git ~/jev-router
cd ~/jev-router
pip3 install -e .

Verify the hook is available:

python3 -c "import jev_router; print(jev_router.__version__)"

Expected output:

0.1.x

Create the Router Directory

mkdir -p ~/ai-router/{config,memory,logs}

Base LiteLLM Configuration

Create ~/ai-router/config.yaml. This file defines the model pool and the master key.

model_list:
  - model_name: local-llama
    litellm_params:
      model: ollama_chat/llama3.2
      api_base: http://localhost:11434

  - model_name: local-mistral
    litellm_params:
      model: ollama_chat/mistral:7b
      api_base: http://localhost:11434

  - model_name: nvidia-deepseek
    litellm_params:
      model: nvidia_nim/deepseek-ai/deepseek-v4.1-flash
      api_key: os.environ/NVIDIA_NIM_API_KEYS

  - model_name: nvidia-glm
    litellm_params:
      model: nvidia_nim/z-ai/glm-5-3
      api_key: os.environ/NVIDIA_NIM_API_KEYS

litellm_settings:
  callbacks: jev_router.hook:router_hook
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  disable_env_credential_login: true

The callbacks line registers the JEV router's pre-call hook. The master_key line enforces authentication. The disable_env_credential_login line prevents the Admin UI from accepting the default environment-based login.

Export the master key:

echo 'export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"' >> ~/.zshrc
source ~/.zshrc

Create a Virtual Key for OpenClaw

Start the proxy manually for key generation:

cd ~/ai-router
litellm --config config.yaml --host 127.0.0.1 --port 4000

In another terminal, generate a virtual key:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["local-llama", "local-mistral", "nvidia-deepseek", "nvidia-glm"],
    "metadata": {"client": "openclaw"}
  }'

The response contains the virtual key. Store it with the client. Generate separate keys for other tools.

Stop the proxy with Ctrl+C once the key is saved.

Router Configuration

The JEV router ships with two decision engines: JevDecider and RulesDecider. The choice is automatic. If TYPESAFE_API_KEY is set, JevDecider runs and JEV makes the routing call. If the key is not set, RulesDecider runs instead. It picks the cheapest eligible model from the candidate pool, filtered by capability flags like vision, output length, and tool support.

For a local-first setup, the default is RulesDecider. No key, no third-party call, no data leaving the Mac mini.

Ensuring Privacy: Unsetting `TYPESAFE_API_KEY`

RulesDecider runs only when TYPESAFE_API_KEY is not present. If the variable is set anywhere — shell profile, .env file, or launchd plist — the router will use JevDecider instead and send minimized summaries to TypeSafe.

Verify the variable is not set:

echo "TYPESAFE_API_KEY=${TYPESAFE_API_KEY:-unset}"

Expected output:

TYPESAFE_API_KEY=unset

If it prints a value, remove it from your shell profile:

sed -i '' '/TYPESAFE_API_KEY/d' ~/.zshrc
source ~/.zshrc

Check any .env files:

grep -R "TYPESAFE_API_KEY" ~/jev-router 2>/dev/null

Check the launchd plist:

grep -A 10 "EnvironmentVariables" ~/Library/LaunchAgents/com.litellm.router.plist

If the variable is listed, remove it. Reload the service if it was running.

Configure the Routing Policy

Create ~/ai-router/router.yaml. This file defines the candidates, their capabilities, prices, and the fallback.

candidates:
  - name: local-llama
    description: Fast local model for simple tasks.
    price: 0.0
    capabilities:
      vision: false
      tools: true
      max_output: 4096

  - name: local-mistral
    description: Balanced local model for everyday coding.
    price: 0.0
    capabilities:
      vision: false
      tools: true
      max_output: 8192

  - name: nvidia-deepseek
    description: NVIDIA NIM model for complex reasoning and long context.
    price: 0.0001
    capabilities:
      vision: true
      tools: true
      max_output: 131072

  - name: nvidia-glm
    description: NVIDIA NIM model for multimodal and agentic tasks.
    price: 0.0001
    capabilities:
      vision: true
      tools: true
      max_output: 131072

fallback: local-mistral

Prices are relative. RulesDecider picks the cheapest model that meets the request's capability needs. Local models have price 0.0, so they win for any request they can handle. NVIDIA models only get selected when the request needs vision, tools, or a context window larger than the local models support.

Switching to Another Routing Path

The endpoint stays http://localhost:4000/v1/chat/completions. The model name stays jev-router. Only the decision layer changes.

Path 2: Local Classifier replaces RulesDecider with a small Ollama model that reads the prompt and returns a tier.

model_list:
  - model_name: jev-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classifier_type: llm
        classifier_llm_config:
          model: ollama_chat/qwen2.5:3b
          api_base: http://localhost:11434
          timeout_ms: 2000
        classifier_fallback: heuristic
        tiers:
          SIMPLE: local-llama
          MEDIUM: local-mistral
          COMPLEX: nvidia-deepseek
          REASONING: nvidia-glm
        complexity_router_default_model: local-mistral

Pull the classifier model first:

ollama pull qwen2.5:3b

Path 3: Semantic Routing replaces the classifier with local embeddings.

model_list:
  - model_name: jev-router
    litellm_params:
      model: auto_router/semantic_router
      semantic_router_config:
        embedding_model: ollama/nomic-embed-text
        routes:
          - name: simple
            utterances:
              - "what is"
              - "define"
              - "hello"
            model: local-llama
          - name: complex
            utterances:
              - "step by step"
              - "analyze this architecture"
              - "refactor the auth module"
            model: nvidia-deepseek
        default_model: local-mistral

Pull the embedding model first:

ollama pull nomic-embed-text

Memori Configuration

Memori must be configured in BYODB (Bring Your Own Database) mode to keep data local. This is done by passing a database connection directly to the Memori constructor. When you provide a valid connection, Memori switches to BYODB mode and sets config.cloud=False internally. Without this connection, it defaults to Memori Cloud.

Verify SQLite Is Available

macOS ships with sqlite3 pre-installed. Verify it is available:

sqlite3 --version

Expected output (example):

3.43.2 2023-10-10 13:00:00

For the latest version with extension support, install it via Homebrew:

brew install sqlite

Homebrew's sqlite is keg-only, so it does not overwrite the system binary. Add it to PATH if you want sqlite3 to resolve to the newer version:

echo 'export PATH="/opt/homebrew/opt/sqlite/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

The system version at /usr/bin/sqlite3 remains available if needed. For the VACUUM; command used below, either version works.

Create the Local SQLite Database

The database file must exist before Memori can build its schema inside it. Create the directory and initialize an empty SQLite file:

mkdir -p ~/ai-router/memory
sqlite3 ~/ai-router/memory/memori.db "VACUUM;"

Expected output: none. Verify the file was created:

ls -lh ~/ai-router/memory/

Expected output:

-rw-r--r--  1 YOUR_USERNAME  staff  4.0K Jan  1 12:00 memori.db

An empty SQLite file is 0 bytes until the first write. The VACUUM; statement forces SQLite to write the file header, so the file appears immediately. If you skip it, the file will be created on the first Memori write.

You can inspect the database at any time:

sqlite3 ~/ai-router/memory/memori.db ".tables"

Expected output before Memori builds its schema: none. After the schema is built:

memori_conversations  memori_memories

Use the Database in the Script

Create ~/ai-router/agent.py. The script points Memori at the SQLite file created above, builds the schema, and enables memory:

import os
from memori import Memori
from litellm import completion
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

# 1. Absolute path to the local SQLite database
db_path = os.path.expanduser("~/ai-router/memory/memori.db")

# 2. SQLAlchemy engine and session bound to that file
engine = create_engine(f"sqlite:///{db_path}")
SessionLocal = sessionmaker(bind=engine)

# 3. Initialize Memori in BYODB mode.
# Passing conn=SessionLocal sets config.cloud=False and config.byodb=True
memori = Memori(
    conn=SessionLocal,
    conscious_ingest=True,
)

# 4. Build the schema inside the local database (idempotent)
memori.config.storage.build()

# 5. Enable memory interception for LiteLLM calls
memori.enable()

# 6. Every completion call now reads and writes to the local database
response = completion(
    model="jev-router",
    messages=[{"role": "user", "content": "What did we discuss last time?"}],
    api_base="http://localhost:4000",
    api_key=os.environ["LITELLM_MASTER_KEY"],
)

print(response.choices[0].message.content)

The key line is conn=SessionLocal. That is what switches Memori out of cloud mode. Without it, the SDK defaults to Memori Cloud and sends data to api.memorilabs.ai.

Verify the Database Is Being Used

After running the script once, check that Memori wrote to the local file:

sqlite3 ~/ai-router/memory/memori.db ".tables"

Expected output:

memori_conversations  memori_memories

Count the rows:

sqlite3 ~/ai-router/memory/memori.db \
  "SELECT COUNT(*) FROM memori_conversations;"

Expected output after one run:

If the count is zero, Memori is not writing locally. Confirm the conn parameter is set and the schema build ran without error.

Verifying Memori Stays Local

Configuration alone is not a guarantee. The SDK contains code for Memori Cloud endpoints, and a documented "cloud tether" issue may still POST to the cloud even in BYODB mode.

Quick audit with lsof:

sudo lsof -i -n | grep ESTABLISHED

Look for entries where the COMMAND is python. If you see connections to IPs that are not your local Ollama (127.0.0.1:11434) or NVIDIA NIM (integrate.api.nvidia.com), that warrants investigation.

Deeper inspection with tcpdump:

sudo tcpdump -n -i en0 'udp port 53'

While this runs, execute a test request through the Memori-enabled agent. If you see DNS lookups for domains like api.memorilabs.ai, the tether is active.

Block at the network level:

echo "127.0.0.1 api.memorilabs.ai" | sudo tee -a /etc/hosts

If the tether is active, the request will fail locally instead of leaving the machine.

Running as a launchd Service

The Mac mini runs 24/7. LiteLLM should start on boot and restart if it crashes.

Create a Wrapper Script

Instead of putting the entire command and all environment variables directly into the plist, create a small shell script that runs the router. This keeps the plist short and avoids formatting issues.

Create ~/ai-router/start_router.sh:

#!/bin/bash
export NVIDIA_NIM_API_KEYS="nvapi-xxxxxxxxxxxxxxxxxxxx"
export LITELLM_MASTER_KEY="sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

cd /Users/YOUR_USERNAME/ai-router
exec /usr/local/bin/litellm --config config.yaml --host 127.0.0.1 --port 4000

Make it executable:

chmod +x ~/ai-router/start_router.sh

Create the launchd Plist

Create ~/Library/LaunchAgents/com.litellm.router.plist:

cat > ~/Library/LaunchAgents/com.litellm.router.plist << 'EOF'




    Label
    com.litellm.router
    ProgramArguments
    
        /Users/YOUR_USERNAME/ai-router/start_router.sh
    
    RunAtLoad
    
    KeepAlive
    
    StandardOutPath
    /Users/YOUR_USERNAME/ai-router/logs/litellm.log
    StandardErrorPath
    /Users/YOUR_USERNAME/ai-router/logs/litellm.error.log


EOF

Replace YOUR_USERNAME with the macOS short username.

Restrict access to the plist and the wrapper script:

chmod 600 ~/Library/LaunchAgents/com.litellm.router.plist
chmod 700 ~/ai-router/start_router.sh

Load the service:

launchctl load ~/Library/LaunchAgents/com.litellm.router.plist

Verify:

launchctl list | grep litellm

Expected output:

12345   0   com.litellm.router

The first number is the PID. The second is the last exit status. 0 means success.

Testing the Full Stack

Send a request that should route locally:

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "jev-router", "messages": [{"role": "user", "content": "Say hello"}]}'

Expected output (truncated):

{
  "model": "local-llama",
  "choices": [{"message": {"content": "Hello! How can I help?"}}]
}

Confirm the routing decision in the logs:

tail -n 5 ~/ai-router/logs/litellm.log

Expected output (example):

INFO: request model=jev-router, resolved=local-llama, provider=ollama

Send a request that requires a large context window:

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "jev-router", "messages": [{"role": "user", "content": "Analyze this 50-page document..."}]}'

Expected output (truncated):

{
  "model": "nvidia-deepseek",
  "choices": [{"message": {"content": "..."}}]
}

Confirm the routing decision in the logs:

tail -n 5 ~/ai-router/logs/litellm.log

Expected output (example):

INFO: request model=jev-router, resolved=nvidia-deepseek, provider=nvidia_nim

The response's model field should report nvidia-deepseek or nvidia-glm, depending on the capability filter.

Confirming the Routing Rule Is Working

Send a request that needs vision (an image input). With vision: true only on the NVIDIA models in router.yaml, the router must skip the local models:

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-router",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]
    }]
  }'

The response's model field must be nvidia-deepseek or nvidia-glm. If a local model appears, the vision capability flag in router.yaml is set incorrectly, or the capability check is not detecting the image input.

Confirming Memori Is Recording

After a few requests, check the local database:

sqlite3 ~/ai-router/memory/memori.db \
  "SELECT COUNT(*) FROM memori_conversations;"

The count should increase with each conversation. If it stays at zero, Memori is not intercepting the calls. Confirm memori.enable() ran before the first completion() call.

Troubleshooting

litellm: command not found — The binary is not in PATH. Use the full path in the plist, or install with pipx.

Service exits immediately after boot — Check ~/ai-router/logs/litellm.error.log. A missing PATH or an unreachable Ollama daemon is the usual cause.

ModuleNotFoundError: No module named 'jev_router' — Install both with the same pip3.

401 Unauthorized — Use a virtual key, not the master key, for client requests.

Requests always route to NVIDIA — Local models may not be listed in router.yaml or may be priced higher than NVIDIA models.

Memori database not created — The SQLite path must be absolute. Replace ~ with the full home directory path.

Memori writes but no rows appear — Confirm memori.config.storage.build() ran before the first completion call. Check sqlite3 ~/ai-router/memory/memori.db ".tables" for the expected schema.

sqlite3: command not found — macOS ships with sqlite3 at /usr/bin/sqlite3. If the command is missing, the system installation has been modified. Install via Homebrew with brew install sqlite and add /opt/homebrew/opt/sqlite/bin to PATH.

Suspicious outbound connections from Python — Use lsof -i -n | grep ESTABLISHED to identify the remote address. If Memori is the source, confirm BYODB mode is active and the /etc/hosts block is in place.

Log file is empty — Confirm the plist writes to the expected path. The StandardOutPath and StandardErrorPath keys must be absolute paths the launchd user can write to.

Happy building.

How to Set Up NVIDIA API Keys and Use Them with Ollama and OpenClaw on a Mac Mini (Part 1)

Haithem Slimi — Mon, 28 Sep 2026 20:36:38 GMT

A Mac mini running Ollama and OpenClaw is a capable local AI stack. The missing piece has always been access to frontier-scale models without needing a GPU cluster. NVIDIA's free API tier on build.nvidia.com fills that gap. It offers open-weight models like DeepSeek, GLM, Kimi, and Nemotron through an OpenAI-compatible endpoint, no credit card required, up to 40 requests per minute.

This guide shows how to obtain an NVIDIA API key, run a local proxy that translates Ollama and OpenAI calls to NVIDIA NIM, and configure OpenClaw to use NVIDIA models directly.

Before You Start: Ollama and OpenClaw Setup

This guide assumes Ollama and OpenClaw are already installed, running, and configured on the Mac mini. If they are not, the foundation is covered in the previous guide: I Turned My Mac Mini Into a Local AI Workstation — Here's Exactly How.

That guide covers:

Installing Ollama and pulling local models.
Installing OpenClaw and setting up its agentic workflow.
Running both as background services on the Mac mini.

What Is NVIDIA NIM?

NIM stands for NVIDIA Inference Microservices. It is NVIDIA's packaging of optimized model containers, exposed through a free API at https://integrate.api.nvidia.com/v1. The endpoint follows the OpenAI Chat Completions protocol, so any OpenAI-compatible client can use it after a base-URL change.

Step 1: Create an NVIDIA API Key

Go to build.nvidia.com and create a free developer account.
Navigate to the API Keys page.
Generate a new key. It will start with nvapi-.

On the Mac mini, export the key in the shell profile so it persists across sessions. For Zsh (default on modern macOS):

echo 'export NVIDIA_NIM_API_KEYS="nvapi-xxxxxxxxxxxxxxxxxxxx"' >> ~/.zshrc
source ~/.zshrc

For Bash:

echo 'export NVIDIA_NIM_API_KEYS="nvapi-xxxxxxxxxxxxxxxxxxxx"' >> ~/.bash_profile
source ~/.bash_profile

The proxy accepts multiple keys as a comma-separated list. For a single key, the format above is sufficient. To verify the variable is set:

echo $NVIDIA_NIM_API_KEYS

Security note: Never hardcode this key into files committed to Git. Export it in your shell profile instead.

Step 2: Run the Ollama Proxy

Ollama does not natively speak to NVIDIA's cloud endpoint. The jjb8966/ollama-proxy project bridges the gap. It is a Flask-based API gateway that routes requests from Ollama-compatible clients to multiple LLM providers, including NVIDIA NIM, using provider-specific prefixes.

The proxy exposes three API shapes simultaneously: Ollama (/api/chat, /api/tags), OpenAI (/v1/chat/completions, /v1/models), and Anthropic Messages (/v1/messages). Requests are routed to NVIDIA NIM when the model name starts with nvidia-nim:.

This section assumes Ollama is already installed and running as described in the previous Mac mini local AI workstation guide.

Clone the Repository

git clone https://github.com/jjb8966/ollama-proxy.git
cd ollama-proxy

Install Dependencies

The project requires Python 3.11 or later.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configure Environment Variables

The proxy reads two environment variables. Both should be exported in your shell profile, same as the NVIDIA API key.

NVIDIA_NIM_API_KEYS — already set in Step 1.
PROXY_API_TOKEN — a secret token you choose yourself. It is not provided by NVIDIA or the proxy. Clients must present this token in their requests to use the proxy. Think of it as a local password that protects your proxy from unauthorized access on your network. Choose any strong, unique string. You can generate one with openssl rand -hex 32.

Add the proxy token to your shell profile. For Zsh:

echo 'export PROXY_API_TOKEN="your-proxy-token-here"' >> ~/.zshrc
source ~/.zshrc

For Bash:

echo 'export PROXY_API_TOKEN="your-proxy-token-here"' >> ~/.bash_profile
source ~/.bash_profile

The proxy also supports Google, OpenRouter, Akash, and Cohere, but only the NVIDIA variables are needed for this setup.

Run the Proxy

python ollama_proxy.py

The proxy listens on port 5002 by default. It can be changed with the PORT environment variable.

Keep the Proxy Running with launchd

A Mac mini that runs 24/7 should start the proxy automatically. Create a launchd plist:

cat > ~/Library/LaunchAgents/com.ollama.proxy.plist << 'EOF'




    Label
    com.ollama.proxy
    WorkingDirectory
    /Users/YOUR_USERNAME/ollama-proxy
    ProgramArguments
    
        /Users/YOUR_USERNAME/ollama-proxy/.venv/bin/python
        ollama_proxy.py
    
    EnvironmentVariables
    
        NVIDIA_NIM_API_KEYS
        nvapi-xxxxxxxxxxxxxxxxxxxx
        PROXY_API_TOKEN
        your-proxy-token-here
        PORT
        5002
    
    RunAtLoad
    
    KeepAlive
    
    StandardOutPath
    /Users/YOUR_USERNAME/Library/Logs/ollama-proxy.log
    StandardErrorPath
    /Users/YOUR_USERNAME/Library/Logs/ollama-proxy.error.log


EOF

launchctl load ~/Library/LaunchAgents/com.ollama.proxy.plist

Replace YOUR_USERNAME with the macOS short username, found with whoami. Replace the API key and proxy token with the real values.

Note: Environment variables must be declared inside the plist. launchd does not read the shell profile, so variables set in ~/.zshrc are not available to background services. This is expected and necessary.

Use NVIDIA Models Through the Proxy

Once the proxy is running, clients call it on port 5002 using the nvidia-nim: prefix. The PROXY_API_TOKEN is already exported in the shell profile, so client code reads it from the environment instead of hardcoding it.

Using the Ollama Python library:

import os
import ollama

client = ollama.Client(
    host='http://localhost:5002',
    headers={'Authorization': f"Bearer {os.environ['PROXY_API_TOKEN']}"}
)

response = client.chat(
    model='nvidia-nim:deepseek-ai/deepseek-v4.1-flash',
    messages=[{'role': 'user', 'content': 'Explain MoE in one paragraph.'}]
)

print(response['message']['content'])

Expected output (example):

Mixture of Experts (MoE) is a technique where multiple specialized sub-models, called experts, are combined. A gating network routes each input token to only a small subset of experts, so the model can have a very large number of parameters while keeping computation low. This makes MoE models efficient and scalable for large language tasks.

Or through the OpenAI-compatible route:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5002/v1",
    api_key=os.environ["PROXY_API_TOKEN"]
)

completion = client.chat.completions.create(
    model="nvidia-nim:z-ai/glm-5-3",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}]
)

print(completion.choices[0].message.content)

Expected output (example):

Silicon chips glow, Parallel cores hum softly, Numbers dance in light.

List all available models with:

curl -H "Authorization: Bearer $PROXY_API_TOKEN" \
  http://localhost:5002/api/tags

Useful model IDs include:

nvidia-nim:deepseek-ai/deepseek-v4.1-flash
nvidia-nim:z-ai/glm-5-3
nvidia-nim:z-ai/glm-5-3-flash
nvidia-nim:moonshotai/kimi-k3
nvidia-nim:nvidia/nemotron-3-ultra-550b-a55b

Model IDs follow NVIDIA's catalog naming. Confirm availability with GET /v1/models on the endpoint if a call returns 404. The proxy's model list is defined in its models.json file.

Why Use a Proxy Alongside OpenClaw?

OpenClaw can talk to NVIDIA directly. So why add a proxy at all? Because the Mac mini is more than just OpenClaw.

Ollama is the home base for local models on the machine. Many tools already know how to talk to Ollama. They use its API. They do not know about NVIDIA. Changing every tool to support NVIDIA separately would be a lot of work.

The proxy solves that. It sits in front of NVIDIA and looks like Ollama to the rest of the system. Any tool that already uses Ollama can now use NVIDIA models by changing only the model name. No new SDK. No new provider code.

This keeps one simple setup:

Local models for quick tasks, private work, and offline use.
NVIDIA models for heavy reasoning, long context, and multimodal jobs.
Same API for both.

The proxy also keeps the NVIDIA API key in one place. Tools do not each need their own key. That is easier to manage and safer.

OpenClaw still uses its native NVIDIA provider for the best agentic experience. The proxy is for everything else. If OpenClaw is the only tool running, the proxy is not needed. If other Ollama-based tools are running, the proxy makes them cloud-capable with almost no effort.

The previous Mac mini local AI workstation guide already sets Ollama as the central hub. The proxy extends that hub to cloud models without disrupting the rest of the setup.

Step 3: Integrate NVIDIA with OpenClaw

OpenClaw natively supports NVIDIA as a model provider. It auto-enables when the NVIDIA_API_KEY environment variable is set and defaults to the Nemotron 3 Ultra model.

This section assumes OpenClaw is already installed and configured as described in the previous Mac mini local AI workstation guide. The NVIDIA API key was already exported in Step 1, though OpenClaw expects it under a different variable name. Set it:

echo 'export NVIDIA_API_KEY="nvapi-xxxxxxxxxxxxxxxxxxxx"' >> ~/.zshrc
source ~/.zshrc

Then run the onboarding command:

openclaw onboard --auth-choice nvidia-api-key

Then set the default model:

openclaw models set nvidia/nvidia/nemotron-3-ultra-550b-a55b

If OpenClaw runs as a background service on the Mac mini, ensure the environment variable is available to the service. If set in ~/.zshrc, that only applies to interactive shells. For launchd services, add it to the plist's EnvironmentVariables section, similar to the proxy setup above.

Manual Config Snippet

To edit the OpenClaw config manually, use this YAML structure:

env:
  NVIDIA_API_KEY: "nvapi-xxxxxxxxxxxxxxxxxxxx"

models:
  providers:
    nvidia:
      baseUrl: "https://integrate.api.nvidia.com/v1"
      api: "openai-completions"

agents:
  defaults:
    model:
      primary: "nvidia/nvidia/nemotron-3-ultra-550b-a55b"

This config auto-loads NVIDIA's featured model catalog from assets.ngc.nvidia.com and caches it for 24 hours, so new models appear without an OpenClaw update.

Use NVIDIA Models in OpenClaw

Switch models interactively within OpenClaw:

/model nvidia/nvidia/nemotron-3-ultra-550b-a55b

Or set it as the default for agentic workflows:

openclaw agents set-default-model nvidia/nvidia/nemotron-3-ultra-550b-a55b

Nemotron 3 Ultra is a 550B total parameter model with 55B active and a 1M-token context window. It is built for long-context agentic work, which suits OpenClaw's multi-step reasoning and tool-calling tasks. Lighter alternatives in the built-in fallback catalog include:

nvidia/nvidia/nemotron-3-super-120b
nvidia/nvidia/llama-3.1-nemotron-70b-instruct
meta/llama-3.3-70b-instruct
nvidia/mistral-nemo-minitron-8b-8k-instruct

The Hybrid Stack

With these changes, the Mac mini runs a hybrid AI stack:

NVIDIA API key is exported in the shell profile and available to background services.
Ollama proxy runs on port 5002 via launchd, providing access to NVIDIA NIM models for Ollama-based tools.
OpenClaw uses NVIDIA NIM directly for agentic coding sessions, with Nemotron 3 Ultra as the default model.
Local Ollama models remain available on localhost:11434 for quick tasks, privacy-sensitive work, and offline use.

Local models handle speed and privacy. Cloud models handle scale and capability. These two paths are separate for now.

Rate Limits and Scaling

The free tier allows 40 requests per minute for most models. For prototyping and individual use, that is sufficient. If the limit is reached, options include:

Wait and retry — the limit resets quickly.
Deploy a NIM container locally on a GPU-equipped machine.
Use a partner endpoint like AWS, Azure, or GCP, which host NIM containers.

Troubleshooting

"Connection refused" on the proxy: Ensure the proxy is running (launchctl list | grep ollama) and that NVIDIA_NIM_API_KEYS is set in the same shell session.

401 Unauthorized from the proxy: The Authorization: Bearer header is missing or the PROXY_API_TOKEN does not match. Include the header in every client request.

401 Unauthorized from NVIDIA: The NVIDIA API key may have expired or been revoked. Generate a new one at build.nvidia.com.

404 Model not found: Check the exact model ID against the proxy's models.json file or NVIDIA's catalog with GET /v1/models. The nvidia-nim: prefix is required for the proxy, and the nvidia/ prefix is required for OpenClaw model references.

Proxy exits immediately after boot: Check ~/Library/Logs/ollama-proxy.error.log. A wrong interpreter path or a missing WorkingDirectory is the usual cause.

Happy building.

NVIDIA Is Giving Away Access to Powerful AI Models for Free

Haithem Slimi — Sun, 27 Sep 2026 15:35:15 GMT

Imagine you want to use a smart AI model to help you write code, summarize a long document, or answer questions. Normally, you either need a very expensive computer or you pay a company for every time you use their AI.

NVIDIA is now offering a third option: you can use some of the world's most powerful AI models for free through their online API. No expensive hardware, no monthly subscription. All you need is an internet connection and a free account.

The Free Models You Can Use

NVIDIA has opened free access to a catalog of open-weight AI models through its developer platform at build.nvidia.com. This means you can use these models without paying for tokens or requests. Some of the models available include:

DeepSeek V4.1 Flash – good for fast reasoning and coding tasks.
GLM 5.3 – a very large model designed for long coding sessions and tool use.
GLM 5.3 Flash – a faster, lighter version of GLM.
Kimi K3 – a massive model built for long conversations and coding.

There are also other models like Nemotron and MiniMax, and the catalog keeps growing.

These are not small, stripped-down demo versions. Kimi K3, for example, is a huge model with billions of parameters. Running it on your own computer would require a server room full of expensive hardware. NVIDIA hosts everything on their own powerful machines.

How to Get Started

The process is simple and free. Here is what you do:

Create a free account at build.nvidia.com.
Find a model with the "Free Endpoint" label. These are the ones you can use without paying.
Generate an API key by clicking your profile icon, going to API Keys, and creating a new key.
Start sending requests to the API.

The API is OpenAI-compatible, which means if you already know how to use OpenAI's API, you can use NVIDIA's the same way. You just change the base URL to:

https://integrate.api.nvidia.com/v1

And you use your NVIDIA API key instead of an OpenAI key.

A Simple Example

Let's say you want to ask DeepSeek V4.1 Flash a question. You would send a request like this:

curl --request POST \
  --url https://integrate.api.nvidia.com/v1/chat/completions \
  --header 'Authorization: Bearer YOUR_NVIDIA_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "deepseek-ai/deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "Write a short poem about the ocean."}],
    "max_tokens": 100
  }'

The model will send back a response. You can do this from any programming language, or even from a tool like Postman.

What You Can Do With It

Here are some practical things you can build:

A coding helper that suggests fixes for your code.
A document summarizer that reads long articles and gives you the key points.
A chatbot for your website that answers common questions.
A study assistant that explains difficult topics in simple terms.

You can also connect these models to coding tools like Cursor or OpenCode, which have built-in NVIDIA integration.

A Few Things to Know

The free tier runs on a rate limit of about 40 requests per minute for most models. This means you can make up to 40 requests every minute. For personal projects, learning, and small experiments, this is usually more than enough.

If you need more, you can always look at the "Deploy" section on NVIDIA's website to run the model on your own infrastructure or through a cloud partner.

There is also a free credit allowance for new accounts, which gives you some extra room before you ever need to think about limits.

Final Thoughts

NVIDIA's free API catalog is a practical way to experiment with powerful AI models without spending money or buying hardware. It is not a fully managed service, and the free tier has limits, but for learning and building small projects, it is a great starting point.

If you have been curious about trying AI models in your own projects, this is a good moment to start.

Try it here: build.nvidia.com

API Base URL: https://integrate.api.nvidia.com/v1

Buzz: The Open-Source Workspace Where AI Joins Your Team

Haithem Slimi — Sun, 27 Sep 2026 15:00:00 GMT

Have you ever worked on a group project? You have a group chat, you share files, and everyone has a role. Now imagine one of your teammates is an AI. It can read the chat, help with code, and do tasks. That is the idea behind Buzz.

Buzz is an open-source project from Jack Dorsey’s company, Block. It is not a magic robot that runs a whole company. It is a tool that lets people and AI agents work together in one place.

An AI agent is a computer program that can do tasks on its own—like reading code, writing a message, or opening a file.

What is Buzz?

Think of it as Slack + GitHub + AI helpers in one app. You can:

Chat with your team.
Share code and files.
Ask AI agents to do work.
See everything in one history.

It uses Nostr, which is a way to give every person and every AI its own digital ID. That ID is like a username that cannot be faked.

A Simple Example

You make a channel called website-fix. You invite:

Your friend Maria (a human).
An AI agent named Fixer.

You write: “The login button is broken.”

Fixer can look at the code, suggest a fix, and open a pull request. Maria can review it. All of this happens in the same chat. You do not need to jump between five different apps.

Key Benefits

AI gets its own account.
It is not just a bot you command. It has a name and an ID. You can see what it did and when.
One place for everything.
Chat, code, tasks, and AI work live together. This saves time.
Works with different AI models.
You can use Claude, ChatGPT, or other AI tools. You are not locked into one company.
You can run it yourself.
Because it is open source, you can host it on your own computer or server. You control your data.
Clear history.
Every action is saved. It is like a group chat log, but it also includes code changes.

What Buzz Is Not

It is not a fully automatic company. Humans still need to lead and make decisions.
It is not a replacement for all your current tools yet. It is early.
It is not perfect. The project is still growing.

Some viral posts say it can run a company “100% by AI.” That is not what it is. It is a workspace for humans and AI to work together.

How to Try It

Go to the GitHub page and read the README:

https://github.com/block/buzz

You can download the app for Mac, Windows, or Linux. Start small: create one channel and invite one AI agent. See how it feels.

Final Thoughts

Buzz is interesting because it treats AI like a teammate, not just a tool. It is a way to explore how humans and AI can work together in one shared space. Ignore the hype about “100% AI companies.” The real value is collaboration.

Security and Governance: The Guardrails That Make AI Safe to Use

Haithem Slimi — Wed, 23 Sep 2026 16:17:12 GMT

Artificial intelligence offers seemingly endless possibilities. It can write, design, diagnose, forecast, negotiate, and increasingly act on our behalf. The potential feels unlimited: faster decisions, lower costs, new services, and solutions to problems once thought unsolvable.

But as AI moves from answering questions to taking action, that potential introduces a new kind of risk. A bad output is inconvenient. A bad action can cost money, expose data, disrupt operations, or cause harm. The question is no longer only "What can AI do?" but "What should AI be allowed to do, and how do we keep it safe?"

The answer lies in two complementary ideas: security and governance. Together, they form the guardrails that make it possible to move quickly without crashing.

What Are AI Guardrails?

A guardrail does not stop a car from moving. It keeps the car on the road when something goes wrong. The same applies to AI.

AI guardrails are the combination of technical controls and organizational rules that keep AI systems safe, accountable, and aligned with human intent. They rest on two pillars.

Security: the lock. Security asks: Can someone misuse this? It protects the AI system from being manipulated, attacked, or broken—and protects people and organizations from the consequences when AI fails.

Governance: the rulebook. Governance asks: Should this be allowed, by whom, and how do we know? It covers policies, approvals, human oversight, audit logs, and accountability.

Neither works alone. Security without governance is a locked door with no rules about who gets a key. Governance without security is a rulebook sitting next to an open vault.

	Security	Governance
Focus	Protection	Direction
Question	Can someone misuse this?	Should this be allowed, by whom, and how do we know?
Analogy	The lock and alarm	The rulebook and referee
Examples	Access controls, encryption, monitoring	Policies, approvals, audit logs, oversight

Guardrails are not barriers to innovation. They are what make innovation sustainable, because they allow AI to be trusted with greater responsibility.

Security: Protecting AI and Protecting from AI

Security focuses on one question: Can someone misuse this? As AI gains access to databases, money, and machines, the attack surface expands. Common risks include:

Prompt injection — tricking the AI into ignoring its instructions.
Data leakage — exposing private or sensitive information.
Unauthorized access — gaining entry to the AI system or its data.
Model theft — stealing the AI model itself.
Supply chain vulnerabilities — compromised third-party tools, libraries, or data.

Security controls reduce these risks. Access controls ensure only authorized users and systems can interact with the AI. Encryption protects data at rest and in transit. Least privilege limits the AI to the minimum permissions it needs. Monitoring and logging record important actions for review. Regular security testing finds vulnerabilities before attackers do.

Security is the lock on the door and the alarm on the wall. It does not decide what the AI should do. It ensures only the right actors get in, and that anything unusual is detected and recorded.

Governance: Rules, Roles, and Accountability

Governance decides what the system is allowed to do. It answers: Should this be allowed, by whom, and how do we know?

An AI system can be perfectly secure and still cause harm. It can make unfair decisions, violate regulations, or act beyond its scope. Governance provides direction, boundaries, and oversight.

Core elements include:

Policies and acceptable use — written rules about what AI can and cannot be used for.
Approval workflows — defined steps for who approves an AI system and which actions require sign-off.
Human-in-the-loop — a human reviews and confirms the AI's recommendation before high-consequence decisions take effect.
Audit logs and traceability — every important action is recorded, enabling investigation and accountability.
Risk classification — systems are sorted by risk level, with stricter controls applied where stakes are higher.
Compliance and ethics — ensuring AI meets legal, regulatory, and ethical standards, including privacy, fairness, and transparency.

Governance is the rulebook and the referee. It does not play the game. It makes sure the game is played fairly and according to agreed rules.

How Security and Governance Work Together

Each pillar covers what the other cannot.

Security without governance is a hardened system with no policy defining what it should or should not do. It remains protected from outsiders, but it can still make harmful or non-compliant decisions.

Governance without security is a detailed rulebook with no enforcement. The rules exist, but the system can be tricked or hacked into ignoring them.

Together, they create safe autonomy: an AI system that is protected from misuse and operates within clearly defined boundaries. This is the foundation of trust, and trust is what allows AI to move from controlled experiments to real-world impact.

Real-World Applications

Customer service AI answers questions, processes refunds, and escalates complex issues. Security protects against prompt injection and unauthorized access to customer records. Governance sets refund limits and requires human approval above a threshold. Every action is logged.

Healthcare AI assists with diagnosis and treatment recommendations. Security protects patient data and restricts access to authorized staff. Governance requires a licensed clinician to approve any AI-generated diagnosis or treatment plan, with audit trails recording who saw what and when.

Self-driving systems control steering, acceleration, and braking. Security protects vehicle software from being hacked and sensor data from being spoofed. Governance sets safety standards, operational limits, and incident reporting requirements, with regulations defining responsibility when something goes wrong.

Enterprise AI agents automate workflows such as scheduling, expense approval, and supply chain management. Security applies least privilege and monitors for unusual activity. Governance defines spending limits, approval chains, and which actions require human confirmation.

Finance and banking uses AI for fraud detection, credit scoring, loan approvals, wealth management, and customer service. It can also act as an agent that moves money or rebalances portfolios. Security protects against data breaches, unauthorized transactions, and model theft. Governance is especially strict here: credit-scoring and loan-approval AI is classified as high-risk under regulations like the EU AI Act, making human oversight, data governance, and explainability mandatory. For AI agents that move money, governance sets predefined mandates, and every proposed action is validated and recorded before it executes. Audit logs are required for compliance. The AI can act, but only within approved, monitored, and recorded rules.

In every case, security protects the system and the data. Governance defines what the AI may do, who approves it, and how we know what happened.

Frameworks for AI Security and Governance

Frameworks are structured guides—recipes, not rigid rules—that organizations can adapt to their own needs.

NIST AI Risk Management Framework (AI RMF) is a voluntary framework built around four functions: Govern (establish roles and policies), Map (understand where AI is used and what could go wrong), Measure (assess risks consistently), and Manage (reduce risks and monitor results). It also defines what trustworthy AI looks like: valid, safe, secure, accountable, transparent, explainable, privacy-enhanced, and fair. It is free, sector-agnostic, and widely used as a starting point.

ISO/IEC 42001 is the first international certifiable standard for an AI Management System. Where NIST offers guidance, ISO/IEC 42001 allows formal certification, demonstrating to customers, regulators, and partners that AI is managed responsibly. It applies to any organization that builds, buys, or uses AI.

OWASP GenAI Security Project maintains the Top 10 for LLM Applications, updated annually based on real-world incident data. The 2026 edition ranks prompt injection, sensitive information disclosure, and excessive agency as the top three risks. A companion Top 10 for Agentic Applications focuses on what autonomous systems are allowed to do once they move from reasoning to action.

EU AI Act is a binding regulation that classifies AI systems by risk level. Creditworthiness and credit-scoring systems are explicitly high-risk, triggering requirements for human oversight, data governance, transparency, and conformity assessment.

MAS SAFR (Safeguards for Agentic Finance at Runtime) was developed by the Monetary Authority of Singapore for AI agents in financial services. It introduces governance checkpoints that verify and record an agent's proposed actions before they execute, keeping behavior within predefined mandates and risk boundaries. It has been applied to agent-assisted payments, wealth management, and client engagement.

These frameworks differ in scope and formality, but they share a core idea: AI should operate within clearly defined boundaries, with oversight, transparency, and accountability. Start with one and build from there.

Practical Checklist

Inventory your AI tools and data access. Know what systems are in use, what data they can reach, and what actions they can take.

Apply least privilege. Give each system only the minimum permissions it needs.

Log all important actions. Record what the AI does, when, and who or what initiated it.

Require human approval for high-risk tasks. Define which actions are too consequential to automate fully.

Test for prompt injection and data leakage. Probe your systems regularly and fix what you find.

Write a simple AI use policy. Document what is allowed, what is prohibited, who approves new use cases, and how incidents are reported.

Review and update regularly. Capabilities, risks, and regulations evolve quickly. What was safe six months ago may not be safe today.

Security and governance are ongoing practices, not one-time projects.

Conclusion

AI offers seemingly endless possibilities, and those possibilities are growing. But as AI moves from answering questions to taking action, the risks grow with it.

Security and governance are the guardrails that make it possible to explore those possibilities safely. Security protects AI systems from being hacked, tricked, or misused. Governance defines what AI is allowed to do, who approves it, and how we know what happened. Together, they create safe autonomy—the ability to trust AI with greater responsibility because the boundaries are clear and the protections are in place.

Frameworks like the NIST AI RMF, ISO/IEC 42001, and the OWASP Top 10 for LLM Applications provide practical blueprints. Real-world examples from customer service, healthcare, self-driving systems, enterprise agents, and finance show that guardrails are already being applied in production systems today.

Trust leads to adoption. Adoption leads to value. Guardrails are what make that journey possible.

Fast Brain, Slow Brain: A Beginner's Guide to System 1 and System 2 AI

Haithem Slimi — Wed, 23 Sep 2026 05:41:50 GMT

You ask an AI "is this email spam?" and it writes you a 200-word essay explaining its reasoning, citing signals, and hedging its conclusion.

Your brain doesn't work that way. You'd just know.

In September 2026, a model called Jev hit 140,000 waitlist signups in 36 hours by doing exactly that: no essays, just decisions. This article explains what it is, why it matters, and when you'd actually use it.

The Two Systems of the Mind

In his book Thinking, Fast and Slow, psychologist Daniel Kahneman described two modes of thought.

System 1 is fast, automatic, and silent. You're walking on a trail and see a curved shape in the grass. Before you can form the word "snake," your body has already flinched. No reasoning. No explanation. Just a decision.

System 2 is slow, deliberate, and verbal. You're doing your taxes. You read the instructions, consider each deduction, check your arithmetic. It's effortful and it feels like thinking.

Here's the mapping that matters for AI:

Today's famous models — ChatGPT, Claude, Gemini — are artificial System 2. They think by writing, one token at a time. That's why they're brilliant at reasoning and terrible at reflexes.
A new category of models aims to be artificial System 1: instant, silent, typed decisions.

Why This Matters Now

Most AI calls in production aren't creative questions. They're judgments:

"Which team should handle this support ticket?"
"Is this transaction risky?"
"Should this agent be allowed to run this tool?"

Right now, we pay System 2 prices — 10 to 40 seconds and cents per call — for System 1 jobs that should take milliseconds and fractions of a cent.

To be fair, classifiers and rerankers have existed for years. What's new is packaging a frontier-quality decision model as a hosted API that anyone can call with a credit card.

And the adoption has been fast. When Jev launched on September 15, 2026, roughly 13% of Vercel's paid AI Gateway teams were using it within 24 hours — more than double any previous model launch.

What Is a System One Model?

Simple definition: unstructured input goes in, typed probabilistic decisions come out. No text generated.

Think of a restaurant host. A party walks in. The host doesn't write an essay about seating theory. They glance at the room and make a decision — instantly, silently, and in one of three shapes:

Decision shape	What the host does	What it returns
Choice	Picks one option from a fixed list	"Booth section"
Score	Rates something on a scale	"Wait time: 0.7 out of 1"
Noul	Answers a yes/no question with confidence	"Reservation on file: yes, 0.95"

That's the whole idea. The host never explains. They never write a paragraph. They just return a typed answer your system can act on directly.

Yes, LLMs can also output JSON. System One models are different in what they're trained to do — calibrated decisions, not just formatted text. Vercel's own comparison notes that Jev "evaluates each question independently against the same state, returning typed answers with probabilities over the defined outcomes," while LLM probabilities are "generated estimates."

The first public example is Jev by TypeSafe AI. You can call it on OpenRouter with the model ID typesafe/jev-1.13, or through Vercel AI Gateway as typesafe-ai/jev.

So How Much Faster and Cheaper, Really?

Independent testers ran Jev against a comparable LLM on the same decision tasks:

Metric	Jev	Comparable LLM
Median latency	105 ms	710 ms
Cost per 1,000 decisions	$0.04	$0.16
Accuracy (vendor dashboard)	67.8%	74.1%

Read that table carefully. Jev is faster and cheaper by an order of magnitude — and slightly less accurate. TypeSafe's own claims of 20–200x speed and 40–400x cost reduction are self-tested against agreement with other frontier models rather than ground truth, and the one independent check (Every) found roughly 25x faster and 580x cheaper on extraction tasks specifically — "good but not perfect."

So here's the honest trade-off. Jev isn't more accurate than the best large models. It's slightly less accurate, and dramatically cheaper and faster. For most production decisions, that's the right trade. For a legal contract or a medical diagnosis, it isn't.

The framing to remember: you don't hire a decision model because it's smarter. You hire it because most decisions don't need smart. They need fast, cheap, and consistent.

Which Jobs for Which Brain?

Task	System One (decide fast)	System Two (think in words)
"Is this ticket urgent?"	✅ ideal	overkill
Routing requests to the right model or agent	✅ ideal	✅ common today
Agent guardrails ("is this tool call dangerous?")	✅ ideal, but pair with deterministic checks	works, but slow
Fraud, spam, or moderation triage at scale	✅ ideal	too costly
Real-time in-app decisions (games, bidding)	✅ only option	impossible (3–30s latency)
Evaluating agent outputs (LLM-as-judge)	✅ promising	✅ common today
Writing an email, essay, or code	❌ can't generate text	✅
Counting, arithmetic, exact dates	❌ weak	✅ (or use code)
Novel problem, open-ended answer	❌ answers must be predefined	✅

The punchline: the winning architecture is hybrid. System One routes, scores, and guards at high volume. System Two handles the escalated hard cases. Plain code does arithmetic.

The constant trap: Archestra tested Jev on 100 real agent tool calls and found that 79% of calls were harmless and stayed local. A classifier with 75% accuracy is therefore worse than a hardcoded "benign" constant. Accuracy alone is meaningless — you need to know the base rate.

Getting Jev Into Your Stack

There are several ways to call Jev, and the right one depends on whether you want to manage credentials yourself or let a gateway handle them.

Route	Model ID	Best for
TypeSafe direct	`typesafe/jev-1.13`	Tightest control, newest features first
OpenRouter	`typesafe/jev-1.13`	One key for many models
Vercel AI Gateway	`typesafe-ai/jev`	Existing Vercel/Next.js apps, zero data retention
LiteLLM proxy	`typesafe/jev-1.13`	Enterprise routing, cost tracking, multi-provider

If you're already running a gateway

This is where Jev's adoption story gets interesting, because gateways didn't just pass it through — they built on it.

LiteLLM added Jev in v1.103.0-rc and proxies its /systemone endpoint with logging and cost tracking. Your TypeSafe key stays on the proxy; clients only need a LiteLLM virtual key. Spend is logged under the versioned model TypeSafe reports (typesafe/jev-1.13.0).

But LiteLLM went further and uses Jev for two jobs inside the proxy itself:

Context compaction. LiteLLM asks Jev whether older tool results are still relevant. If Jev scores a result below 0.2, it replaces the result with a short notice — saving input tokens on long agent runs.
Auto Router classification. LiteLLM's router uses Jev to decide which model tier a request belongs to. In their benchmark, Jev matched expected tiers on 95% of calls versus 73.75% for Claude Haiku, at 5.43x the speed and 96% lower cost.

Vercel AI Gateway exposes Jev with zero data retention and no output token charge. Vercel's AI SDK ships an experimental evaluate API that surfaces typed answers directly in TypeScript — so a department choice can select a support queue, a severity score can influence priority, and an uncertain result can trigger human review.

Cloudflare Workers AI, Netlify AI Gateway, and OpenRouter all serve Jev as well. Netlify's integration is zero-config: install @typesafe-ai/sdk in a Netlify Function, and AI Gateway handles credentials and billing.

The pattern across all of them is the same: Jev is cheap enough to call inside infrastructure, not just from application code. That's why routers use it as a classifier and proxies use it as a guardrail.

Limitations You Should Know

Typed ≠ correct. A wrong answer from your allowed list is still wrong.

No explanations. That makes audits harder. Keep humans in the loop for high-stakes calls.

Prompt injection is real and vendor-acknowledged. TypeSafe's own limitations page for Jev 1.13 states that "content written to adversarially steer the model, whether that is an injected instruction, a deliberately misleading framing, or text that argues for its own classification, can move the answer." An independent security benchmark found 10 false positives and 13 false negatives out of 662 prompt-injection tests. A guard built on Jev belongs alongside deterministic checks, not instead of them.

Weak at arithmetic, dates, counting, and adversarial input. Leave those to code.

Not a replacement for LLMs. It's a different tool for a different job.

The Takeaway

System Two writes. System One decides. The future is both.

You're running 10 million classifications per day. A decision model gets 29 out of 30 right for a fraction of a cent each. A large LLM gets 30 out of 30 for a hundred times more. Where's your cutoff — and what would you never trust to a decision model?

References

TypeSafe AI — Jev API documentation and limitations page: https://docs.typesafe.ai/api
LiteLLM — "TypeSafe Jev on LiteLLM" (Sep 20, 2026): https://docs.litellm.ai/blog/typesafe\_jev
LiteLLM — "Reduce agent context with TypeSafe Jev and LiteLLM" (Sep 18, 2026): https://docs.litellm.ai/blog/typesafe-jev-compaction
LiteLLM — "JEV Classifier: 5.43x as Fast as Haiku, 96% Lower Cost" (Sep 20, 2026): https://docs.litellm.ai/blog/jev-classifier
Vercel — "How to classify, route, and score with Jev and AI SDK": https://vercel.com/kb/guide/typesafe-jev-and-ai-sdk
CryptoBriefing — "TypeSafe opens Jev AI to public after rapid adoption forces waitlist removal" (Sep 21, 2026): https://cryptobriefing.com/typesafe-jev-ai-public-access/
explainx.ai — "Is Jev's 200x-Faster, 400x-Cheaper Claim Actually True?" (Sep 19, 2026): https://www.explainx.ai/blog/jev-speed-cost-claims-fact-check-2026
Archestra — "We Tested Jev on 100 Real Agent Calls. How Easy Is It To Beat a Constant?" (Sep 21, 2026): https://archestra.ai
VentureBeat — "Jev AI agent security: Prompt injection risk" (Sep 21, 2026): https://venturebeat.com
Layer3Labs — "Jev Benchmarks: How Accurate Is TypeSafe AI's Model?" (Sep 22, 2026): https://www.layer3labs.io
Daniel Kahneman — Thinking, Fast and Slow (2011)

Memory for AI Agents: Architectures, Tools, and How to Choose the Right One

Haithem Slimi — Sun, 20 Sep 2026 13:53:16 GMT

Imagine talking to someone who forgets everything the moment you stop speaking. Every conversation starts from zero. No context. No learning. No continuity.

That’s how most AI chatbots work today. They’re stateless—they have no memory of past interactions. But “agentic AI” is changing that. An agent is an AI that can take actions, use tools, and pursue goals over time. To do that well, it needs memory—just like a human.

This piece breaks down the memory problem for AI agents, the main architectural patterns, the tools available, and the criteria you should use to choose the right one. At the end, I’ll share a recommendation that stands out for production systems.

1. Why AI Agents Need Memory

Without memory, an AI agent:

Repeats mistakes.
Forgets user preferences.
Loses context across sessions.
Can’t learn from its own actions.
Feels like a stranger every time you talk to it.

Memory turns a chatbot into a colleague. It lets the agent remember what was said, what was done, and what worked. That’s the foundation of agentic AI.

2. The Four Types of AI Memory (In Simple Terms)

Most frameworks borrow from human cognition and split memory into four types:

Type	What It Holds	Example
Working Memory	The “right now”	Current conversation, active task
Episodic Memory	“What happened”	Past conversations, actions, outcomes
Semantic Memory	“Facts and knowledge”	User preferences, world facts
Procedural Memory	“How to do things”	Learned skills, successful action sequences

A good agent memory system needs to handle all four—or at least the ones relevant to its job.

3. Architectural Patterns for Agent Memory

There’s no single way to build memory. Here are the main patterns, from simplest to most advanced:

a) Simple In-Process Memory

Everything stays in the current conversation window. Easy, but the agent forgets once the session ends.

b) External Vector Store

The agent saves memories in a database and retrieves relevant ones using semantic search. Like a searchable notebook.

c) Tiered Memory Architecture

Layered storage: “hot” (fast, recent), “warm” (less frequent), and “cold” (archival). Balances speed and capacity.

d) Knowledge Graph Memory

Memories are stored as connected nodes (entities and relationships). Helps the agent reason about how facts relate.

Each pattern has trade-offs in cost, latency, complexity, and accuracy.

4. The Tool Landscape: A Quick List

Here are the most talked-about memory tools for AI agents today:

Mem0 – Plug-and-play conversational memory. Good for simple use cases.
Letta (formerly MemGPT) – Pioneered agents that edit their own memory.
Zep – Focuses on temporal reasoning—how memories change over time.
LangChain / LangMem – Memory components inside a broader agent ecosystem.
Memori – A SQL-native, structured memory layer that remembers actions, not just chat.

Each has strengths. The right choice depends on your specific requirements.

5. How to Choose a Memory Tool

Before picking a tool, ask:

What does it remember? Just chat, or also tool calls and outcomes?
Where is memory stored? Proprietary vector store, or your own database?
Can you audit it? Can you inspect and query memories directly?
How much does it cost? Tokens per query matter at scale.
How fast is it? Does memory creation slow down the agent?
Can you deploy it your way? Cloud, on-prem, or bring your own database?
Does it forget intelligently? Stale memories should fade; important ones should stay.
Does it work with your inference stack? If you run local models, does the tool support them natively?

These criteria will guide you to the right tool for your use case.

6. Evaluating the Options: Why Memori Stands Out

After applying the criteria above, one tool consistently rises to the top for production agents: Memori.

Memori isn’t just another memory tool. It treats memory as a data structuring problem, not a text-stuffing problem. Here’s why that matters:

✅ SQL-Native and Auditable

Memori uses everyday databases like SQLite, PostgreSQL, or MongoDB. You can inspect, query, and audit memories directly. No black-box vector store.

✅ Remembers Actions, Not Just Chat

Most tools capture conversation. Memori also captures tool calls, execution paths, decisions, and outcomes. The agent learns from what it does, not only what it says.

✅ Structured Memory

It converts messy dialogue into clean, queryable facts and relationships. That makes recall more precise and less token-heavy.

✅ Intelligent Recall + Decay Scoring

Memories that are accessed often and recently get prioritized. Stale ones fade away. This keeps context relevant without manual cleanup.

✅ Performance and Cost

On the LoCoMo benchmark (a standard long-context memory test), Memori reportedly achieved 81.95% accuracy—outperforming Mem0 (~~62%), LangMem (~~78%), and Zep (79%). And it did this using only **1,294 tokens per query**, roughly 5% of the cost of stuffing the full conversation into the prompt.

✅ Deployment Flexibility

Use Memori Cloud, or bring your own database (BYODB) for full control. Great for enterprise, on-prem, or VPC deployments.

✅ Native Ollama Integration — A Platform Capability, Not an Afterthought

Memori is LLM-agnostic by design, and local OpenAI-compatible endpoints like Ollama, vLLM, and llama.cpp are treated as first-class providers. This matters because a growing share of agent development happens on local or self-hosted inference, yet many memory layers assume a cloud LLM.

Memori supports Ollama through three concrete mechanisms:

Universal provider support. Memori ships with tested examples for OpenAI, Azure OpenAI, LiteLLM, and Ollama. Its “any OpenAI-compatible” configuration path means most local inference servers work out of the box.
LiteLLM as a bridge. Ollama is supported natively via LiteLLM, so the same memory pipeline that talks to GPT-4 or Claude can also talk to a locally running Llama or Qwen model without code changes.
Local embeddings stay local. Memori’s retrieval layer supports Ollama embeddings (e.g., nomic-embed-text), so the entire memory pipeline—extraction, augmentation, and recall—can run entirely offline.

For anyone building a local-first agent stack, this is the difference between a memory layer that can work with your inference engine and one that does work with it by design.

7. Multi-Agent Memory: Shared, Not Distributed

When you have more than one AI agent, a key question is: what should they share, and what should stay private?

Memori uses a shared memory database with clear rules about what each agent can see. Think of it like a shared notebook for all your agents, but with different sections that are locked or open.

There are three main levels of separation:

👤 User Level (Shared Across All Agents)

All agents that serve the same user share basic facts and preferences. For example, if you tell your support bot that you use PostgreSQL, your sales bot can later say, “I see you use PostgreSQL—here’s a product that works well with it.” The sales bot doesn’t need to ask you again.

What’s shared at this level:

Facts (e.g., “Alice uses PostgreSQL”)
Preferences (e.g., “Alice prefers email over phone”)
Skills (e.g., “Alice knows Python”)
Knowledge graph (connections between facts)

🤖 Agent Level (Private to Each Agent)

Each agent has its own private notes about how it works. The support bot and the sales bot each have their own attributes and conversation histories. The sales bot doesn’t see the full chat log from the support bot.

What’s private at this level:

Attributes (e.g., the support bot’s tone settings)
Conversation history (each agent keeps its own record of what was said)

💬 Session Level (Private to Each Conversation)

Even within the same agent, each conversation session is kept separate. If you start a new chat with the support bot, it remembers you use PostgreSQL (from the shared user memory), but it doesn’t remember the exact words from your previous chat session.

What’s private at this level:

The specific messages exchanged in that session
The context of that particular conversation

A Simple Example

Imagine you have two agents: a personal assistant and a shopping helper.

You tell your personal assistant: “I love dark roast coffee.”
Later, you ask your shopping helper: “What coffee should I buy?”
The shopping helper can say: “Since you love dark roast, here are some options.”
But the shopping helper cannot see the entire conversation you had with the personal assistant—only the fact that you love dark roast.

That’s the balance: shared knowledge, private conversations.

Summary Table

What is it?	Who can see it?	Example
Facts & Preferences	All agents for the same user	“Alice uses PostgreSQL”
Skills	All agents for the same user	“Alice knows Python”
Attributes	Only the specific agent	Support bot’s tone settings
Conversation history	Only the specific agent + session	The exact messages in one chat

So Memori is not a peer-to-peer system where each agent has its own memory that syncs with others. It’s a central shared memory with smart rules about who sees what. That makes it easier to build multi-agent systems where agents collaborate without stepping on each other’s toes.

8. When Memori Might Not Be the Best Fit

Memori is powerful, but it’s not for everyone. If you need a quick, plug-and-play memory for a simple chatbot, Mem0 might be easier. If you want an agent that rewrites its own memory in a research setting, Letta is interesting. But for production agents that need structured, auditable, cost-efficient memory across many sessions and multiple agents, Memori is a very strong candidate.

9. The Big Takeaway

Memory is not just a feature. It’s a fundamental architecture problem.

For AI agents to be truly useful over long periods, they need a well-designed memory system that can store, retrieve, and forget information intelligently. Stop treating memory as an afterthought. Design it as a first-class part of your agentic AI system.

And if you want a memory layer that’s structured, SQL-native, action-aware, cost-efficient, Ollama-compatible, and shared across multiple agents with smart isolation—Memori is a very strong place to start.

References

Cognitive architecture for AI memory. “Your Model Has Humanity’s Cortex. It Needs Its Own Hippocampus.” Medium. https://medium.com/codetodeploy/your-model-has-humanitys-cortex-it-needs-its-own-hippocampus-36a4b9676c8d
Local AI workstation setup. “I Turned My Mac Mini Into a Local AI Workstation — Here’s Exactly How.” Hashnode. https://nightthoughts.hashnode.dev/i-turned-my-mac-mini-into-a-local-ai-workstation-here-s-exactly-how
Memori overview. Memori Labs. “Introducing Memori Cloud: Fully Hosted SQL-Native Memory Layer for AI Agents.” https://memorilabs.ai/blog/launching-memori-cloud/
Memori multi-user isolation model. Memori Docs. “Multi-User Support.” https://memorilabs.ai/docs/memori-cloud/concepts/multi-user-support/
Memori LoCoMo benchmark results. Memori Labs. “Releasing Our LoCoMo Benchmark Paper.” https://memorilabs.ai/blog/memori-locomo-paper-results/
Memori LLM provider support (Ollama, vLLM, llama.cpp). Memori Manual — Doramagic. https://doramagic.ai/en/projects/memori/manual/
Memory tool comparison (Letta, Mem0, Zep, LangMem). LoCoMo Benchmark Results. Hugging Face. https://huggingface.co/datasets/rovemark/locomo-benchmark-results/blob/main/README.md

Note: Benchmark numbers are based on Memori’s published results. Always verify with your own use case before choosing a tool.

What Is LLM Routing? A Simple Guide to Saving Money on AI

Haithem Slimi — Sun, 20 Sep 2026 03:46:20 GMT

Most teams pick one AI model and use it for everything.

A short "summarize this" request costs the same as a complex "debug this code" request. That's wasteful — but the waste is invisible because your bill doesn't tell you which requests were easy.

Think of it like taking a taxi for every trip. Even a trip to the corner store. A router is like choosing between a bike, a bus, or a taxi depending on the trip.

This guide explains what routing is, why it matters, and what tools exist. No code, no math. Just plain ideas.

What Is a Gateway? What Is a Router?

These two words get mixed up all the time. Here's the difference.

Gateway = the front door. It handles logins, API keys, logging, and sending your request to the right place.

Router = the decision-maker inside the front door. It looks at your request and decides which model should handle it.

Failover = if the chosen model is down or slow, the gateway tries another one automatically.

A router is like a GPS that reroutes you when there's traffic. It's not just picking the fastest route — it's picking a route that still works when the main road is closed.

Most gateways include a router. Some routers work on their own.

The Hidden Risk of One Model

Routing isn't just about cost. It's about staying online.

In 2026, every major API provider has had outages. OpenAI, Anthropic, Google, and AWS have all gone down for hours. If your app depends on one model, your app goes down when that model does.

A local model on your own hardware is not immune either. Your machine can crash, overheat, or lose power.

The fix is the same as the cost fix: have more than one model, and a layer that can switch between them.

Teams that added routing for cost reasons discovered it also saved them during outages. Teams that added it for resiliency discovered it also cut their bill. Either reason is enough.

The Tricky Part: Context Sharing

Routing sounds simple. Send easy requests to cheap models, hard requests to expensive ones.

But there's a catch most beginners don't see coming: models don't share memory.

Imagine switching drivers in the middle of a race. The new driver gets in the car and asks, "Where are we going? What happened so far?" If nobody tells them, they're lost.

That's what happens when you switch models mid-conversation.

Every model has a limit on how much it can remember at once. This is called the context window. Some models remember a lot. Some remember very little. If your conversation is too long for the new model, it either fails or forgets the beginning.

Here's a real example. You ask your local model to "fix this function and run the tests." It calls a tool. The tool result comes back. Then you switch to a cloud model for the next step. The cloud model has no idea what function you're talking about, what tests ran, or what failed.

Three simple rules to avoid this:

Don't switch models in the middle of a task. Finish the task, then switch if needed.
Keep conversations short when switching. Long chats lose more context.
If you must switch, pass a short summary. Even one sentence like "We're fixing a Python function that sorts a list" helps a lot.

There's also a money angle. When you stay on the same model, providers give you a discount on repeated text. Switch models, and you lose that discount. So switching isn't just a context problem — it can cost more too.

For a personal setup like a Mac Mini, you don't need to solve all of this. Just know it exists. The simplest fix is: pick one model per task, and stick with it until the task is done.

The Four Approaches

Each approach has a cost benefit and a resiliency benefit. Cascade gives you both in one design.

Cascade — Try a cheap model first. If it fails or the answer seems weak, try an expensive one. Simple to set up. The expensive model is already your fallback. Best for beginners.

Semantic — Guess how hard the request is before sending it. Easy requests go to cheap models, hard ones go to expensive models. More accurate than cascade, but needs a classifier. You configure failover separately.

Cost-Aware — Predict how much the request will cost before sending it. Pick the cheapest model that can handle it. Best for budget control. But pure cost focus means no built-in fallback.

Learned — Train a small AI model to make routing decisions. Highest ceiling, but needs data and machine learning skills. Resiliency depends on whether your training data includes failures.

Approach	Cost	Resiliency
Cascade	High	High — built in
Semantic	High	Medium — needs setup
Cost-Aware	Highest	Low — no fallback
Learned	High	Medium — depends

The Tools — What's Actually Out There

Here's what exists in 2026. Split into two categories: gateways and routers.

Gateways

Tool	Cost	Failover	Best For
LiteLLM	Free OSS (self-hosted)	Yes	Control on a budget
OpenRouter	5.5% fee on top-ups	Yes	Fastest start
Portkey	Free 10K req/mo; Pro $49/mo	Yes	Governance
Cloudflare AI Gateway	Free tier; pay beyond	Yes	Apps on Cloudflare
Kong AI Gateway	Free OSS core; from ~$500/mo	Yes	Enterprises on Kong

Routers

Tool	Cost	Failover	Best For
RouteLLM	Free OSS	No	Simple cost routing
vLLM Semantic Router	Free OSS + GPU	Yes	High-volume serving
NVIDIA Switchyard	Free OSS	Yes	Agent workflows

Most teams need a gateway with routing built in, not a standalone router. LiteLLM is the most common starting point because it's free, open source, and includes routing.

The Numbers — And Why They Vary

Savings numbers are real but come from specific setups. Here's what the data actually shows.

Source	Savings	Setup	Shows
LiteLLM production	51% → 60.7%	450+ users, 270k requests	Savings grow with tuning
LiteLLM production	95% served by non-flagship	Auto-Router classification	Most requests are easy
Subtask routing	46% cheaper	Coding agent	Routing by task type works
AT&T production	Up to 90%	45B tokens/day	Scale amplifies savings

Why the numbers vary:

Your workload matters. If most of your requests are complex, routing saves less.
Your model pair matters. Routing between two cheap models saves little. Routing between cheap and expensive saves a lot.
Tuning matters. The LiteLLM deployment started at 42.9% savings and climbed to 60.7% over four months as the team adjusted which requests went where.

Routing is not magic. It works when you have a mix of easy and hard requests. If every request is hard, it won't help.

What's Next

In the next piece, we'll add a routing layer to the Mac Mini setup. Not just to save money — but to keep working when the cloud goes down.

We'll install LiteLLM, connect local and cloud models, set up failover, and measure what it actually saves on a real workload.

References

LiteLLM. "51% Cost Savings Reported From a Live Production Deployment." LiteLLM Blog, August 10, 2026.
LiteLLM. "Subtask-Specific Routing: Same Quality, 46% Less Cost." LiteLLM Blog, September 7, 2026.
Contabo. "Best LLM Gateways in 2026: Top LiteLLM Alternatives." Contabo Blog, June 12, 2026.
OpenRouter. "Pricing." OpenRouter, 2026.
Portkey. "Feature Comparison." Portkey Docs, July 27, 2026.
Mobile World Live. "Analysis: AT&T bets on routing, open model to tame AI costs." Mobile World Live, July 28, 2026.

Open-Weight vs Proprietary AI: Cost, Licensing, and Accuracy Compared

Haithem Slimi — Sat, 19 Sep 2026 16:04:22 GMT

Everyone is talking about open-weight AI models. You can download them. You can run them. You can even build products on them.

So they must be free, right?

Not even close.

This study breaks down what AI models actually cost in 2026 — split into open-weight and proprietary models — and shows why "open" is not the same as "free."

But before we dive in, let's understand the basic unit of AI pricing: the token.

Before We Start: What Is a Token?

AI models don't read words like humans do. They break text into small pieces called tokens.

A token is roughly 4 characters or about ¾ of a word in English.

Text	Tokens
"Hello"	1
"Artificial intelligence"	2
"ChatGPT is amazing"	4
"I love learning about AI models"	7

When you use an AI model through an API, you pay per 1 million tokens. There are two types:

Input tokens — the text you send to the model
Output tokens — the text the model sends back

Output tokens almost always cost more than input tokens — usually 5 to 6 times more. Generating new text takes more computing power than reading existing text.

So when you see "$1.25 input / $10 output per 1M tokens," you pay $1.25 for every million tokens you send, and $10 for every million the model generates. Long answers get expensive fast.

Now that you understand tokens, let's look at the real costs.

Part 1: Open-Weight Models Are Not Free

Open-weight models are models whose parameters are publicly downloadable. Anyone can grab them.

But three hidden costs come with them:

Hardware. Big models need big machines.
Licenses. Many "open" licenses have strings attached.
People. Someone has to run and maintain the servers.

The License Trap

Model	License	Commercial Use & Catch
DeepSeek R1 / V3	MIT	Free, no restrictions
Gemma 4	Apache 2.0	Free, no restrictions
Phi-4	MIT	Free, no restrictions
GLM-5.2	MIT	Free, no restrictions
OLMo	Apache 2.0	Free, no restrictions
Llama 4	Llama Community	Free until 700M monthly users
Mistral Medium 3.5	Modified MIT	Commercial supply restriction
Kimi K3	Custom	Revenue share up to 30% for MaaS
Grok-2	Custom	Free only under $1M/year revenue
StableLM	Custom	Free only under $1M/year revenue
Command R	CC-BY-NC	Research only. No commercial use

The lesson: MIT and Apache 2.0 licenses are genuinely free. Custom licenses are not. They look open, but they're really "free until you get big."

The Hardware Bill

Model	Size	Hardware Needed	Rough Cost
Kimi K3	2.8T params	Two 8-GPU H200 servers	$320K–$420K+
Llama 4 Maverick	400B	8× H100 GPUs	$80K–$120K
DeepSeek V3	671B	4–8 datacenter GPUs	$50K–$80K
Gemma 4 31B	31B	One gaming GPU	~$2,000
Phi-4	14B	One gaming GPU	~$2,000

Small models really can be nearly free. Frontier models cannot.

The Break-Even Point

Renting an 8-GPU NVIDIA H100 node on a specialized cloud provider costs roughly $18,000 per month, though major hyperscalers can charge over $70,000 for the same setup.

Daily Usage	API Cost/Month	Self-Host Cost/Month	Cheaper
1M tokens	$150	$18,221	API
50M tokens	$7,500	$18,221	API
100M tokens	$15,000	$18,221	API
500M+ tokens	—	—	Self-host

Simple rule: Unless you're processing over 100–500 million tokens per day, self-hosting a big open model costs more than just paying for an API.

Part 2: Proprietary Models — Pay As You Go

Proprietary models are rented, not owned. You send text, you get answers, you pay per token.

Model	Company	Input $/1M	Output $/1M
GPT-5.5	OpenAI	$5.00	$45.00
GPT-5	OpenAI	$1.25	$10.00
GPT-5 mini	OpenAI	$0.25	$2.00
GPT-5 nano	OpenAI	$0.05	$0.40
Claude Opus 5	Anthropic	$5.00	$25.00
Claude Sonnet 5	Anthropic	$2.00	$10.00
Gemini 3.8 Flash	Google	$0.75	$3.75
DeepSeek V4 Pro	DeepSeek	$0.66	$1.98
Kimi K3	Moonshot AI	$3.00	$15.00

The good news: no license negotiations, no user caps, no hardware. You just pay the bill.

Two things to watch:

Output costs more than input — usually 5–6× more.
Batch processing is 50% cheaper if you don't need answers immediately.

Part 3: Cost vs. Accuracy

Here's the big question: how much accuracy do you lose by going cheap?

General Knowledge (MMLU-Pro)

Model	Type	Score	Input Cost
Qwen3.7 Max	Proprietary	89.6%	API only
Claude Opus 4.5	Proprietary	89.5%	$5.00
Qwen3.5 397B	Open weight	87.8%	Self-host
Kimi K2.5	Open weight	87.1%	$3.00
DeepSeek V4 Pro	Open weight	87.1%	$0.66
Gemma 4 31B	Open weight	~85%	Nearly free

The gap is about 2 points — but the price difference is 40×.

Real-World Coding (SWE-bench Verified)

This is the hard one. It measures whether a model can actually fix real software bugs.

Model	Type	Score
Claude Opus 5	Proprietary	96.0%
Claude Fable 5	Proprietary	95.0%
Ornith-1.5-397B	Open weight	86.0%
DeepSeek V4 Pro	Open weight	80.6%
MiniMax M3	Open weight	80.5%
Kimi K2.6	Open weight	~75%

Here the gap is real. For genuinely hard work — complex coding, multi-step reasoning — proprietary models still win.

Part 4: So What Should You Do?

Use open weights when:

Your task is simple (summarizing, sorting, basic chat)
You need data to stay on your own servers
You're processing over 100M tokens/day
You want to avoid vendor lock-in

Use proprietary models when:

The task is hard (complex reasoning, agentic coding)
Your volume is low or medium
You need legal protection against IP claims
You don't want to hire engineers to run servers

The best answer for most teams: use both. Route easy tasks to cheap open models. Send hard tasks to frontier proprietary models.

The Bottom Line

Open-weight models changed the game. They can cut your costs by 80–95% while giving up only a few points of accuracy.

But three things stay true:

Downloading is free. Running is not. A frontier open model can cost $300K+ in hardware.
"Open" licenses often have limits. Only MIT and Apache 2.0 are truly free.
Someone has to maintain it. Budget for engineering time, not just GPUs.

The smartest teams in 2026 aren't choosing sides. They're mixing both.

References

MillionMiner. "AI Server Price Guide 2026: Rent Vs Buy." September 9, 2026.
IntuitionLabs. "Open-Weight AI Model Licenses: Commercial Use Rules Explained." September 5, 2026.
Taskade. "10 Best Open-Source LLMs, August 2026." May 23, 2026.
BenchLM. "Self-Hosting vs API: Break-Even Calculator for Open-Weight LLMs." July 3, 2026.
OpenAI. "Pricing." OpenAI API, 2026.
Anthropic. "Claude Opus 5 Pricing." July 24, 2026.
Google. "What's new in Gemini 3.8 Flash." September 3, 2026.
BenchLM. "MMLU-Pro Leaderboard & Scores — September 2026." September 9, 2026.
BenchLM. "SWE-bench Verified Leaderboard (September 2026)." September 10, 2026.
Meta. "Llama 4 Community License Agreement." April 5, 2025.
Hugging Face. "Kimi K3 License." July 30, 2026.
Reuters. "China's DeepSeek to make permanent 75% price cut on flagship V4-Pro AI model." May 23, 2026.

All pricing and benchmark data reflects publicly available information as of September 2026. Verify current terms and pricing before making procurement decisions.

When AI Eats Itself: Understanding Model Collapse and AI Cannibalism

Haithem Slimi — Fri, 18 Sep 2026 21:30:57 GMT

There is a snake in the AI industry, and it is eating its own tail.

For years, the recipe for building better AI seemed simple: scrape more data from the internet, train a bigger model, repeat. But we have reached a strange turning point. A huge portion of the internet is now written by AI. And the next generation of AI is being trained on that very content. The models are eating themselves.

This is not a metaphor. It has a name, and it is a formally proven problem called model collapse.

What the Researchers Found

In July 2024, a team led by Ilia Shumailov at Oxford University, working with researchers from Cambridge and other institutions, published a landmark paper in the journal Nature. Their finding was stark and simple: if you train AI models on data generated by other AI models, they will break down in an irreversible way.

They called this effect "model collapse".

The study showed that this does not just happen to one type of model. They observed the collapse in large language models (LLMs), in image models called variational autoencoders, and even in simple statistical models called Gaussian mixture models. The problem is baked into the very nature of how these systems learn.

Why Does Collapse Happen?

To understand why, you need to think about what a model actually learns. It is not learning "facts." It is learning statistical patterns from enormous piles of data. It learns what words tend to follow other words, what ideas tend to appear together.

A healthy dataset is full of variety. It contains common ideas that appear all the time. But it also contains rare things: minority languages, unusual turns of phrase, niche knowledge, strange edge cases. Researchers call these rare parts the "tails" of the data distribution.

Here is the problem: every model slightly overproduces the common stuff and underproduces the rare stuff. It is a small bias, but it exists in every generation of the model.

Now imagine you take that output — which is slightly skewed toward the average — and you train the next model on it. The rare content shrinks even more. You do this again, and again. The tails of the distribution disappear entirely.

What you are left with is a model that has forgotten the richness of the real world. It only knows the middle, the average, the bland. Its outputs become repetitive and eventually nonsensical.

The Jackrabbit Problem

The researchers gave a vivid demonstration of this. They took a small model called OPT-125m and fed it a passage of text about 14th-century church towers.

Then they let it generate new text. They trained a new model on that output. Then another. And another. They did this for nine generations.

By generation nine, the model had completely forgotten about architecture. When asked about medieval church towers, it started listing species of jackrabbits — including fictional ones like "blue-tailed jackrabbits".

The model did not just get a fact wrong. It had lost the entire conceptual grounding of the original topic. It was hallucinating entirely.

This Is Happening Right Now

This is not a theoretical problem for the future. It is a problem for right now.

A study from the company Graphite analyzed over 55,000 English-language articles published between 2020 and 2026. They found that in the first quarter of 2026, an estimated 49.9% of sampled articles were classified as mostly AI-generated.

In late 2024, that figure was around 48%. By late 2025, AI-generated articles briefly surpassed human-written ones.

Other studies paint a similar picture. One analysis of over 1.2 million webpages from late 2025 to early 2026 found that roughly 35% contained wholly or partially AI-generated content. Estimates suggest that 30–40% of the active web corpus is now synthetic.

This matters because those same models are trained on web-scraped data. They are increasingly ingesting content that other models wrote. The contamination is not a future scenario. It is the current state of the training pipeline.

Can It Be Fixed?

The research is not all doom. But the fixes are not simple.

The most basic idea is to mix real data with synthetic data. A paper presented at an IEEE conference in late 2025 proposed a simple strategy: combine synthetic data and human-generated data so that the model does not degenerate. They derived mathematical conditions on the ratio of synthetic to human data needed to keep the system stable.

But there is a catch. The same paper notes that on a small GPT model, enriching synthetic data with just a small amount of human data may not be enough to prevent collapse. The balance matters.

Another approach is to use detectors. Researchers at University College London trained a machine-generated text detector and proposed a method to up-sample likely human content in the training data. They found that this not only prevented model collapse but actually improved performance compared to training on purely human data.

A third approach is to monitor entropy. As a model collapses, the entropy of its training data declines. The model stops generating novel content and starts memorizing its training samples. Researchers have proposed using this entropy decline as an indicator of degradation and selecting data to preserve diversity.

A fourth idea is verification. Some researchers have shown that injecting information through an external verifier — whether a human or a better model — can prevent synthetic retraining from causing collapse.

The common thread in all these solutions is the same: human-generated data is irreplaceable. The original paper put it plainly: "the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet".

What This Means for You

If you are building or training models, the lesson is straightforward: know where your data comes from. If you are scraping the web post-2022, you are training on synthetic content. Monitor your model's diversity and entropy during training.

If you are working with data pipelines, understand that genuine human data is becoming a strategic asset. The value of real conversations, original writing, and real-world observations is rising precisely because it is becoming scarce.

And if you are watching the industry broadly, the "free lunch" of scraping unlimited web data is ending. Sustainable AI development will require deliberate investment in data provenance, curation, and verification.

The snake can stop eating its tail. But it has to choose to do so before the tails disappear entirely.

Sources

Shumailov, I. et al. "AI models collapse when trained on recursively generated data." Nature 631, 755–759 (2024).
Graphite study on AI-generated articles, reported by Search Engine Land (2026).
Data Science Dojo, "AI Cannibalism: How Models Are Eating Themselves Into Collapse."
IEEE conference paper, "Preventing Model Collapse when Training LLMs with Synthetic Data" (2025).
UCL research on machine-generated text detection (2025).
NeurIPS paper, "A Closer Look at Model Collapse" (2025).

Cover image source: Data Science Dojo