ContextCaching

Context Caching VS Semantic Memory

Strategy for using AI API

Ria Choi • October 20th, 2026

Context Caching VS Semantic Memory

I am building two projects at the same time. Both have a server composer structure. Each project has its own algorithm based decision server. Each also calls an AI API.

So the server has one big question to answer every time. Should it pass the job to the algorithm server, or call the AI?

I expected both projects to handle AI memory the same way. They didn't.

Floww is an AI agentic wallet. It works best with semantic memory.

Orcha is my personal AI agent schedule manager. It works best with context caching.

Why the difference? Let's find out.

The Goal Decides the Strategy

Orcha is a personal project. My goal was simple: lower the cost.

I did not want to spend many tokens on every question. My app was sending too many rules and policies in each request, and my token budget was disappearing fast.

Floww is the opposite. The goal was a solid and accurate memory. It does not matter how many tokens it uses.

Same stack, different goals. That is the whole story.

What Is Context Caching?

Context caching lets you store a large block of input once. Later requests point to that stored block instead of sending it again.

Think of it like this. Without caching, you hand the AI the full rulebook every time you ask a question. With caching, the AI already holds the rulebook. You only send the question.

I chose the Gemini API for this. It is a standard quality, cheaper option, and my project does not need hard reasoning. Gemini also offers context caching, which solved my token problem.

For Orcha, this works well because:

The rules and policies are mostly fixed.
The same rules are needed in every request.
There is no database call before the AI call.
Repeated tokens cost less.

The rulebook is static. So caching it is the natural choice.

What Is Semantic Memory?

Semantic memory stores long term knowledge. This includes facts, concepts, and user preferences. It does not store every detail of a past event. It stores what was learned from it.

For example, an agent can remember that a user cares about translation quality. It can apply that preference next time.

You can build this in PostgreSQL with pgvector. pgvector is an extension that stores vector embeddings and runs similarity search inside your database.

Here is the basic flow:

AI Agent
   ↓
Embedding Model
   ↓
PostgreSQL
   ├── Regular Tables (memories, users, tasks)
   └── pgvector (embeddings + similarity search)
   ↓
Relevant Memories → AI Agent

One thing to remember. pgvector does not create memories for you. It only stores and searches embeddings. Your app must decide what to save and how to read it.

If you want the full walkthrough, I wrote about it here: How to manage AI Persistent Memory in PostgreSQL.

Why I Did Not Use Semantic Memory for Orcha

Semantic memory has a cost. Every time you ask the AI something, your server must reach the database first.

That means an embedding step, a search, and then the AI call.

This does not solve my token problem. It adds more steps. For a cost saving project like Orcha, it was the wrong tool.

The tool was never the point. Gemini can support both approaches. The strategy is what matters.

So Why Does It Work for Floww?

Floww has different needs. Here is why semantic memory fits.

1. Accuracy comes first.
Floww is a wallet. A wrong rule can cause a real problem. Extra tokens are a fair price for correct behavior.

2. Memory changes over time.
A cached block is a snapshot. Semantic memory can grow with each user and each decision.

3. Only relevant rules are sent.
Search finds the rules that match the current task. The AI gets focused context instead of one giant rulebook.

4. It stays under control.
Memory supports the AI's decisions. It does not override user authorization or backend execution policies. The backend still has the final word.

Here the database call is not a weakness. It is the feature.

How to Choose

Ask yourself two questions.

First, does my context stay the same between requests? If yes, try context caching.

Second, does accuracy matter more than cost? If yes, try semantic memory.

You can also combine them. Cache the stable base rules. Use semantic memory for what changes. I have not done this yet, but it is an option I am considering.