By ByteBot
Your AI agent doesn’t have amnesia by design — it has amnesia by default. Conversation history gets dropped on restart. Tool call logs vanish when the container exits. Vector embeddings live in a separate service your team scrambles to keep in sync. Most production agent failures trace back to state management, not model quality. Databricks Lakebase — now generally available on both AWS and Azure — gives agents a serverless Postgres database that starts in under 500 milliseconds, branches instantly, and feeds your data lakehouse without a single ETL job. The Data+AI Summit opens today (June 9-12), and Lakebase is leading the keynote.
What Lakebase Actually Is
Databricks acquired Neon in 2025 and built Lakebase on its core architecture. Neon’s breakthrough: separating Postgres compute from storage. Your data lives in zone-redundant object storage with NVMe SSD caching. The compute layer is fully stateless and serverless — it scales to zero when idle, resumes in under 500ms, and bills per compute unit (2 GB RAM each).
For developers, none of that complexity surfaces. Lakebase is Postgres. Standard drivers work. pgvector works. pg_stat_statements works. LangGraph’s checkpoint library works. If you can write psycopg2 code, you can use Lakebase. Provisioning happens through the Databricks SDK, so you spin up instances programmatically rather than clicking through a console — which is exactly what agent-driven infrastructure needs.
The Four Things Agents Need to Remember
Agent state falls into four categories, and Lakebase handles all of them in a single database:
- Session context: Current conversation turns, active tool calls in progress
- Long-term memory: User preferences, facts the agent learned across sessions
- Operational logs: Which tools fired, what they returned, execution order
- Semantic vectors: Embeddings for RAG and similarity search via pgvector
This consolidation matters more than it sounds. When agent memory lives in Redis (session), SQLite (logs), and Pinecone (vectors), you have three systems to keep consistent, three failure modes, and three billing relationships. Lakebase collapses all of that into standard Postgres.
Connecting in Python: The Minimal Version
The Databricks SDK handles credential management. Get the connection details from your project instance and connect with psycopg2:
from databricks.sdk import WorkspaceClient
import psycopg2
client = WorkspaceClient()
instance = client.postgres.get_project("agent-memory-db")
conn = psycopg2.connect(
host=instance.read_write_dns,
user=instance.user,
password=instance.password,
sslmode="require"
)For LangGraph users, swap psycopg2 for the langgraph-checkpoint-postgres package and point it at the same connection string — thread-based conversation context becomes durable automatically. For SQLAlchemy, the connection string is postgresql://user:password@host:port/database?sslmode=require. Standard Postgres, no surprises.
Instant Branching Is the Feature Agent Developers Actually Want
Most serverless Postgres providers offer scale-to-zero. Lakebase does something more interesting: zero-copy database branching. Fork a production database with all its data in milliseconds, run an agent test against real conversation history, then discard the branch when done. No staging environment maintenance. No data sync scripts.
For agent development this is transformative. Test a new memory schema against production data without risk. Run multiple agent variants in parallel against isolated forks of the same dataset. Each branch has its own compute allocation — idle branches cost nothing, active ones scale on demand. When the test finishes, the branch disappears and billing stops. Databricks calls this branch-per-agent: one Lakebase project, many branches, each agent’s state fully isolated. Over 80% of databases provisioned on the platform today are created by AI agents programmatically.
Why Lakebase Beats Standalone Neon If You’re Already on Databricks
Here’s the honest take: if you’re building a greenfield agent app with no existing data infrastructure, Neon standalone is faster to start and cheaper for small workloads. Supabase is better if you need auth, realtime, and storage in one stack. Lakebase wins on one specific condition: your operational data needs to talk to your analytics data.
When an agent writes a conversation turn to Lakebase, that row is continuously synced to Delta Lake in Unity Catalog — no pipeline, no Kafka, no Airbyte. Your Spark jobs can query agent conversation logs immediately. Your BI dashboards show live agent activity. Your ML feature pipelines consume operational data without delay. Unity Catalog enforces governance across the whole surface — the same audit trail and data lineage rules that govern your warehouse also govern your agent’s memory.
That integration is genuinely valuable when you’re operating at scale. It’s overkill for a side project. Know which one you’re building.
The Data+AI Summit and What’s Coming Next
Databricks is running the Data+AI Summit in San Francisco this week (June 9-12, Moscone Center). The keynote is expected to feature Lakebase alongside Agent Bricks — Databricks’ framework for connecting agents to platform capabilities. The Azure GA announcement dropped in the week leading up to the summit (now live in 14 Azure regions worldwide). Storage pricing fell significantly post-Neon acquisition: $0.35/GB-month, down from $1.75. If you’re evaluating Lakebase for production use, watching the keynote is worth the time.
Lakebase won’t replace your existing Postgres for every workload. But for production AI agents that need durable, queryable, governed state — and for teams already on Databricks — the answer to “where does my agent remember things?” just got considerably simpler. The database your agent needs already exists. It’s serverless, it branches, and it speaks standard Postgres. You just have to point your connection string at it.
