How to choose mission-critical tools: a Keycloak lesson

In short

For mission-critical systems like authentication, score every option on team familiarity, documentation, support and operational load, not only on features. Run a short, time-boxed proof of concept before you commit. AI makes unfamiliar tools easier to learn and makes building your own far cheaper, but it doesn't remove the need for someone to own the tool in production. And the AI tools themselves deserve the same scorecard.

What happened

At Fincaraiz we were moving a legacy ASP.NET monolith to Django and React microservices, for a platform with more than 5 million monthly visits. The old and new systems had to run side by side, so we needed one identity layer both could share. Keycloak was a sensible choice: open source, full-featured and widely used for single sign-on.

The problem was fit, not quality. Keycloak was new to the team and built on Java, which was outside our core skills. When integration issues appeared, nobody could dig into its internals with confidence, and the documentation had gaps for our exact setup. Every problem became an investigation. That part of the project took six times longer than estimated.

Why it happened

We evaluated the tool, not the fit. Our comparison of features was thorough. What we didn't ask was: when this breaks under real traffic, who on our team can fix it, how long will it take them, and where will they look for answers?

A tool is only as good as your team's ability to run it when it fails.

The scorecard we use now

After an open lessons-learned session, we added a vetting step for any mission-critical tool. Each option gets a simple score from 1 to 5 on these questions:

Criterion	The question to ask
Stack fit	Is it built on a language and platform the team already knows?
Team experience	Has anyone on the team run it in production?
Documentation	Do the docs cover our exact use case, with working examples?
Support	Is there an active community, paid support, or both? How fast do issues get answered?
Operational load	Who patches, upgrades and monitors it, and how much time does that take each month?
Integration effort	How much custom code does it need to work with our systems?
AI fit	How well do AI assistants know it? Is it widely used or open source, with clear public docs, or even an MCP server our assistants can connect to?
Data exposure	What data would the tool, or the AI features built into it, send outside our systems, and are we allowed to send it?

A tool that scores low on stack fit and team experience can still win, but only if the plan includes the time and training to learn it, and a named owner.

The last two rows didn't exist when we chose Keycloak. AI fit matters because a team now learns a new tool partly through its assistants: a widely used tool with good public docs gets accurate help, while a niche one gets confident guesses. Data exposure matters because more tools now ship with AI features that send your data to a model provider, sometimes switched on by default.

Build, host or buy

For identity specifically, the options today fall into two groups, and the right choice depends on your team as much as your requirements:

Managed identity services such as Auth0, Okta, AWS Cognito or Microsoft Entra ID. You trade some flexibility and a monthly bill for far less to operate.
Self-hosted open source such as Keycloak. More control and no per-user licence, but your team runs it, upgrades it and secures it. Keycloak has become easier to operate in recent versions, but the ownership question doesn't go away.

In practice, the deciding factor is operational ownership. Self-hosting an identity provider means owning its availability targets and on-call rotation, security patching and CVE response, version upgrades, database backups, and signing-key rotation. If the team can't commit to that, a managed provider is the safer architecture choice.

How AI assistants change the trade-off

If we faced the same decision today, AI tools would help in three ways, and leave one problem untouched.

Learning an unfamiliar stack is faster. Assistants like Claude Code, Cursor or GitHub Copilot can explain an unfamiliar Java codebase, walk through configuration, and draft integration code. The "we don't know this language" penalty is smaller than it was.
Proofs of concept are cheaper. A working spike can often be built in days instead of weeks, so there is no excuse to skip one for a critical tool.
Documentation gaps are easier to bridge. An assistant can read the source code and the issue tracker when the docs fall short.
Agents can do the first pass of the evaluation. AI coding agents can now install a tool in a sandbox, configure it, write integration tests and report what broke, often in an afternoon. That makes it realistic to run a small proof of concept on two or three options instead of one.
But ownership stays human. AI can help you set a tool up. It can't be on call for it, and for niche configuration it can be confidently wrong. For security-critical systems, every AI-suggested change still needs review by someone who understands it.

A fourth option: build it with AI

Build, host or buy used to have an obvious bias: building was expensive, so you bought or self-hosted. AI coding assistants have changed that. A small internal tool, an integration or an admin screen that once took weeks can now be generated in days.

That makes building tempting, and sometimes right. But generated code has the same owner as handwritten code: you. Someone has to understand it, test it, patch it and keep it running after the person who prompted it moves on.

My rule of thumb:

Kind of system	Build with AI?
Small internal tools, scripts, admin screens, one-off integrations	Often yes. Cheap to build, low risk, easy to replace.
Core product features your team will evolve for years	Yes, if the team reviews and understands every change.
Security-critical infrastructure: authentication, payments, encryption	No. Use a proven product or service. Being cheap to generate doesn't make it safe to own.

Identity is in the last row. AI would have made our Keycloak integration faster. It would not have made writing our own identity provider a good idea.

The AI tools need a scorecard too

The same mistake we made with Keycloak is easy to repeat with AI itself: choosing a model, an assistant or an agent platform because of a demo, without asking who will own it. The scorecard above still applies, with a few additions:

Evaluations. Can we measure whether it works on our real tasks, with a test set, not only in a demo?
Cost at scale. What does it cost per use when the whole team, or every customer, uses it every day?
Change rate. Models and tools change every few months. Can we pin a version, and who checks that an update didn't break anything?
Data handling. Where do prompts and outputs go, are they retained, and are they used for training?
Exit plan. If the provider changes price or terms, how hard is it to switch?

AI tools are moving faster than anything else in the stack, which makes a named owner more important, not less.

A simple process for your next decision

Write down the requirements, including who will operate the tool.
Shortlist two or three options, including at least one the team already knows, and consider whether building it with AI is a real option.
Score each option with the scorecard above, including AI fit and data exposure.
Run a time-boxed proof of concept on the top options, using AI agents to move fast. Ask an assistant to argue against your favourite option, then check its claims.
Decide, name an owner, and record the reasons so the next team understands the choice.

How the team handled it

The overrun could easily have turned into blame. Instead, we treated it as a shared lesson and wrote the scorecard together. After that, people raised risks earlier, which was worth more than any single tool decision.

Read the full story of the migration in Monolith to microservices at Fincaraiz.

Erwin MonguiDirector of Technology at Cedar Planters and Certified AI Engineer, based in Toronto. More about me