AI: What I've Built, and What I Haven't
The AI work I've done directly, organized by capability rather than by project.
Forty years running enterprise technology programs and portfolios. Over the last six months I've also been building and operating AI systems myself, and doing AI work inside an enterprise with security, legal, and privacy in the room. This is what that covers.
Multi-agent orchestration and synthesis
I built a multi-agent advisory council that runs a real decision through five agents with separated roles and distinct lenses. They debate in sequence, peer-review each other anonymously, and a synthesis layer produces a single verdict. Ten-plus completed runs, including one delivered as an executive deliverable in a paid engagement for a firm in a regulated industry.
The design lessons were the point: agents degrade fast when their roles blur, and anonymizing the peer-review step materially changes what comes out the other side.
Evaluation and scoring design
I designed the scoring rubric behind a pipeline that evaluates inbound job postings against a detailed fit model, using an LLM at low temperature for consistency. When my criteria changed, I retuned the rubric and re-scored against known cases to check the change did what I intended.
The rubric and the evaluation loop are the hard part. The pipeline around them is comparatively easy — and tuning an eval loop is the part most people talking about AI have never actually had to do.
Production AI pipelines and scheduled automation
I built and run a production pipeline end to end: API ingestion of source data, LLM scoring, persistence to a local database, filtering, and delivery of a daily shortlist to a messaging channel. It runs unattended every day.
Alongside it, three independently authored scheduled briefing jobs and automated security scans across my own infrastructure. Earlier, a no-code pipeline that watched governance news, had an LLM summarize and tag each item into a structured template, and filed it into a knowledge base. I decommissioned that one within months when better tooling arrived, which is the more useful habit.
Agent infrastructure and LLM application architecture
I installed, configured, and personally operate an always-on agent framework with persistent memory, sub-agent orchestration, and scheduled jobs — running on a Linux environment I stood up, hardened, and back up automatically. It is the operating layer everything above runs on, not a tool I tried once.
I have also designed a full LLM application architecture on paper: a three-layer memory model combining a structured user profile, vector embeddings with semantic retrieval, and progressively compressed conversation summaries to stay inside context limits — plus model tiering to control inference cost, voice pipeline vendor selection, and per-user unit economics. Specified in detail, not built.
Operating AI systems under change control
I maintain a formal regression suite for the agent platform I run — eight layers covering gateway startup, scheduled jobs, messaging adapters, sub-agent execution, configuration drift, provider connectivity, memory integrity, and live user-facing delivery. Every release goes through a soak period before I accept it, and the suite runs against every upgrade.
The most recent run earned its keep. It caught two real regressions — a plugin that silently stayed pinned to the prior version, and a hot reload that reported the new version while still serving the old binary — and separated both from a session-local fault that looked like a failure and wasn't. Every finding goes back into the suite as a new test. Most organizations running agents have no answer here yet.
Applied AI inside an enterprise, under governance
At lululemon I led the technical and security work on an AI video-analytics pilot. I produced the artifacts myself — architecture diagrams, a detailed threat model with actors, trust boundaries, controls and references, a security questionnaire, and an architecture deck that opened with the business case — then represented the program to the global architecture and security teams for approval. Some of it I argued and won. Some of it came back and I reworked it.
In parallel, Legal and Privacy were still defining what responsible AI meant for the company. I worked with them directly as the technical counterpart, pushed back where the emerging standard would not survive contact with a real deployment, and carried it to the vendors who had to accept it. The incumbent video platform signed it as part of a global retrofit. The pilot vendor was presented the same commitment, but the program was cancelled at the executive level before signature and before any store went live.
Separately, in a client engagement, I wrote a closed-AI systems brief for a firm operating under strict client-imposed AI restrictions — separating what “no AI” contract language actually prohibits from what remains available under inference-only, no-retention, BAA-covered, and local-model conditions, with carve-out language and counsel questions attached.
AI portfolio evaluation and spend governance
I ran a structured evaluation of AI tools against specific use cases rather than feature lists, and built a taxonomy of the AI market across fourteen sub-sectors to see where capabilities sit and where they overlap.
I also track my own AI spend against utilization monthly, and route work across Claude, Gemini, and ChatGPT by cost and fit rather than defaulting to one premium model. Tracking it monthly is what lets me tell a real efficiency gain from a simple drop in activity. Unowned, duplicated AI spend is a live problem in most organizations right now, and it is the same intake-and-watch-the-money discipline I have applied to capital portfolios for decades.
AI-assisted delivery
I designed and built randallrusher.ai end to end using AI tooling, with no developer and no designer involved. I have produced client deliverables the same way, including a presentation built from a raw audio recording of a working session.
I have also used agents to produce program artifacts at volume and then governed the output: a catalog of roughly 450 requirements tagged by phase, priority, source, and status, drafted by an agent under my direction and reviewed by a second model for coverage gaps and priority errors, with every correction accepted or rejected by me. AI produces the draft; a person owns what ships.
Randall Rusher · Aruca, LLC