Joseph Stickel

Joseph Stickel

Applied AI Engineer, Production RAG, Agents, Evals & Private LLM Platforms

jobtrue.ai/u/mrjstickel/applied-ai-engineer

Joseph Stickel

Applied AI Engineer, Production RAG, Agents, Evals & Private LLM Platforms

Everett, WA  ·  contact@mrjstickel.com  · 
mrjstickel.com · github.com/j-stickel · linkedin.com/in/mrjstickel

Summary

Built Architecture Zero, a white-label AI platform whose core is open source under Apache-2.0, running live production instances from one codebase. Cleared a pre-registered, per-cohort eval gate on every row (held pair, 2026-10-24: 88.3 tuned vs 76.6 on an outside-authored locked holdout, the 11.8-point gap published, honesty 13/13), graded by an LLM judge from a different model family, calibrated on planted errors (51/51). Orchestrated parallel AI coding agents through production backlogs under proof gates (a test that fails on the old code, every diff read before it lands) and took a 36-finding outside security review to full closure.

Technical Skills

AI / LLM SystemsRAG pipelines (hybrid vector + BM25 fusion), cross-encoder reranking, retrieval eval harnesses, LLM-as-judge answer evals, LLM-judge calibration / meta-evaluation, pre-registered eval gates, agentic tool use with person-only write approval, MCP servers (OAuth 2.1), permission-aware connectors (Google Drive, Microsoft 365), QLoRA / PEFT fine-tuning, multi-provider routing (Claude / GPT / Ollama) with failover, local inference (7B to 30B-class on an RTX 5090), ChromaDB, embeddings, prompt-injection defense, AI-assisted engineering (Claude Code: parallel coding agents in git worktrees, custom skills / hooks / multi-agent workflows)
Security & IdentityOIDC sign-in (Microsoft Entra, Google Workspace), OAuth 2.1 with PKCE, MFA (TOTP), per-document access control, Sign-In-With-Ethereum (EIP-4361), Content-Security-Policy, SSRF and DNS-rebinding defense, sandboxed document parsing, secret scanning and dependency audit gates, OWASP Web + LLM Top 10
BackendPython, FastAPI, Node.js, Firebase Admin, REST API design, Stripe, SQL, MySQL, SQLite, Alembic migrations
InfrastructureGCP (Compute + Cloud Monitoring uptime alerting), Vercel, Cloudflare (DNS, tunnels, R2 object storage), Docker Compose, Nginx, Let's Encrypt, CI/CD security gating (gitleaks, pip-audit, bandit, trivy), BetterStack monitoring
FrontendNext.js, React, TypeScript, Tailwind, React Native, Expo
Real-TimeFirebase RTDB, Upstash Redis, Server-Sent Events, distributed locking

Experience

AI Systems & Platform EngineerJan 2022 - Present

Independent: Architecture Zero, TheStatic.tv, JobTrue · Remote

  • Architected Architecture Zero, a white-label AI platform whose core is now open source under Apache-2.0, running live production instances from one core codebase: hybrid RAG (vector + custom Okapi BM25 + cross-encoder reranker), department-scoped knowledge bases with role-based access, multi-provider LLM routing (Claude / GPT / local Ollama) with failover, and local inference (7B to 30B-class models on an RTX 5090).
  • Cleared a pre-registered, per-cohort eval gate on every row of one held pair (2026-10-24), its thresholds written before the numbers existed: answer correctness 88.3 tuned vs 76.6 on an outside-authored locked holdout (the 11.8-point overfit gap published), honesty 13/13, faithfulness 98.7, retrieval recall 90% (tuned) and 87% (full cohort), injection and hostile-web cohorts held 9 of 9. Answers and grades come from structurally different labs; the thin margins and the one rule change made before the run are reported beside the verdict.
  • Built Eco Mode, a federated retrieval mesh: instances query each other's knowledge in parallel with per-caller scoped keys and fail-closed access boundaries. A public bot can never reach private departments; verified by adversarial testing as the least-privileged caller.
  • Took a 36-finding outside security review of the whole multi-product workspace (8 High) to full closure across three rounds of outside re-checks: every finding verified at its cited line, every fix shipped with a test that fails on the old code, and the reviewer's final check re-ran those tests against the predecessor code. A pre-push read of my own found what the review had not: a production database's client rules let any signed-in account write its own balance and tier. Locked to server-only and audited for any sign of use.
  • Shipped TheStatic.tv solo: a production Web3 streaming platform, 160+ authenticated API endpoints, GCP Livestream RTMP/HLS pipeline, a cron-driven billing engine with Redis distributed locks and viewer-scaled burn rates, and an ERC-721 contract on Ethereum mainnet.
  • Shipped JobTrue (jobtrue.ai) solo: an AI job-search SaaS with Stripe billing, LLM fit-scoring with a server-side grounding layer that drops hallucinated resume items before storage, and per-user rate limiting. This resume is served from it.
  • Operate everything built (hosting, CI/CD, monitoring, security review, documentation) plus nightly encrypted backups to immutable object storage (Cloudflare R2 bucket-lock + Google Drive) and an automated daily restore drill that boots each backup in a throwaway container: 9/9 checks, alarmed end-to-end via a fail-closed status probe.
  • Calibrated the LLM-as-judge eval instrument against four independent references: a planted-error suite (51/51 caught across five rubrics, 2026-10-24), a human-adjudicated baseline (Cohen's kappa 0.61), a cross-lab re-grade (99.0% agreement, kappa 0.942), and an independently-authored harness (98.2% boundary agreement). Then audited my own locked exam, retired it for authorship bias, re-authored it via a third lab, and published the wider honest gap.
  • Designed refuse-vs-fabricate honesty evals (cohorts demanding artifacts the corpus does not hold, scored by a purpose-built judge rubric with planted calibration), measured 12/12 and 13/13; drove an adversarial tier-isolation cohort from 3/11 to 11/11 with owner recall held at 100%. Built a four-arm cohort measuring over-refusal (the cost of a correct answer flipped into an unnecessary refusal), isolating it to 0/6 where it had been recorded as a wholesale refusal, and tracing that result to an uncontrolled test-fixture variable.
  • Rolled the measured trust layer across every production instance, validating each port with like-for-like A/Bs on that instance's own cohort: +12 correctness on one peer, +18.4 recall with rank-1 hits doubled on another, and multi-turn follow-up retrieval driven 1/9 to 8/9 by a measure-gated resolver port.
  • Eliminated hand-set status from the platform's trust dashboards: every number derives from stored eval runs or live runtime config at request time, never from a constant. Each reliability layer publishes either its live-derived measurement or an honest build status, so unmeasured layers stay visibly unmeasured rather than rounded up, and controls that are silent when disabled (rate limiting, injection scanning, egress screening) read their real runtime state instead of asserting "on". The tuned-vs-holdout overfit gap publishes as a first-class metric even when it widens.
  • Root-caused silent production failures behind green health checks: an HNSW vector-index corruption (search dead while reads passed, shipped a knn tripwire plus per-run corpus fingerprinting) and two distinct vector-store data-loss mechanisms (flush-boundary loss on unclean stops; a file-watcher mass-purge during git deploys). Fixed and verified with back-to-back zero-loss deploy cycles.
  • Open-sourced the platform's core under Apache-2.0 as a derivation rather than a fork: every file read in full and landed as a commit stating its keep-or-drop reasoning, on fresh history that structurally cannot leak what it never held. Gated on a six-lens adversarial pre-release review (27 findings, all remediated pre-publish) and a residue hunt for the class literal scrubbing cannot see (document shapes, date-subject pairs, metrics sampled from reality), which found 16 further items after a literal scrub verified clean.
  • Exposed a private AI system to third-party clients read-only over MCP, then shipped an OAuth 2.1 authorization server so a hosted assistant could connect with a real handshake proven server-side. Deliberately zero write or confirm tools in the module, so the REST-only owner-confirmation rule keeps holding on a surface where a model holds the tools.
  • Split every FastAPI entry point in the platform by job (9,825, 7,205, 4,856, 4,798 and 4,269 lines down to 400-650 each, nearly 600 route handlers into routers), every commit proven behavior-identical by snapshot diffs of the built app (each route's auth dependencies, the OpenAPI schema, startup order, middleware) and the full suite, with zero production incidents; then carried the method to the React clients under render snapshots. Found on the way a class of tests a refactor leaves passing while they test nothing, proven vacuous by mutation and re-pointed.
  • Hardened every deployment, public open-source template first: stored credentials encrypted, MFA that fails closed, the caller's own password required at every write that creates durable authority, security state that survives restarts, production SSH closed to an identity-aware tunnel with deploys on pull-based runners, and untrusted document parsing moved into a locked no-network sandbox, its locks proven inside each live container. A cross-repository drift check holds the controls in place, itself fixed after four security fixes reached every private instance and missed the template.
  • Built per-document access control for a RAG knowledge base: a Google Drive connector reads each file's real permissions and enforces them per document at retrieval, evaluated server-side in the vector store so a document the caller may not read is never fetched, never reranked, and never reaches the model's context. Demonstrated rather than asserted: a full administrator with no grant on a document retrieves it by neither the chat path nor the corpus browser, while the granted identity retrieves all of it.
  • Built a platform's self-serve support layer as retrieval-optimized documentation (one self-contained answer per chunk-aligned section, troubleshooting entries keyed to the exact error strings the code emits so a pasted error retrieves its own explanation) and made supportability measurable: the shipped evaluation seed targets the help corpus with expected sources, so a fresh install scores its own onboarding and support answer quality on its first evaluation run.
  • Orchestrated parallel AI coding agents through production backlogs, runs of 42, 60 and 84 agents, each agent in an isolated git worktree under a fact-checked brief and a fixed proof standard (a test that fails on the old code), and integrated every result myself: the diff read at its lines, the full suite and a browser harness of up to 2,000 checks after each landing. The 84-agent run put 361 commits into production; the 42-agent run, 260 commits through six gated deploys with zero regressions. Security defects the bug lists never named were found by the agents' own reads.
  • Priced a cross-encoder reranker against its alternative with a cohort-matched A/B over 93 questions on the live corpus, adding the ordering metrics the harness lacked (nDCG@k, MRR) because recall cannot distinguish the answer at rank 1 from rank 5: the reranker bought +7.6 points of recall and +0.10 nDCG for +28.3 seconds per query, which kept it in the system and priced the hardware decision instead of guessing at it. A free read-only probe run first rejected the obvious cheap fix (cutting the candidate pool would have spent ~10 points of recall ceiling), so nothing shipped to learn it.
ProctorFeb 2026 - Present

TEEMA · Everett, WA

  • Maintain operational availability of testing workstations for aerospace manufacturing candidate assessments with a zero-downtime requirement.
  • Serve as primary on-site contact for all technical issues; resolve hardware, peripheral, and connectivity problems and escalate to remote engineering when needed.
  • Enforce strict site security and technical protocols as the on-the-ground liaison for off-site infrastructure managers.
Technical Support SpecialistSept 2025 - Feb 2026

Decentraland Foundation · Remote

  • Provided technical support for creators and developers building on the Decentraland metaverse platform across Intercom, Discord, and Slack.
  • Troubleshot SDK issues, scene deployment problems, wearables, and platform bugs; documented reproductions and coordinated with engineering teams for resolution.
  • Leveraged 4+ years of hands-on Decentraland development experience to give expert-level guidance to the creator community.
Technical Support Team Lead / Support Engineer IINov 2022 - Jun 2025

Symetrix, Inc. · Mountlake Terrace, WA

  • Drove live call answer rate from under 25% to over 80% by rebuilding NetSuite case-routing workflows and KPI tracking as primary NetSuite administrator.
  • Cut Return Authorization turnaround from several weeks to 1-2 weeks by redesigning the process mid-2024.
  • Hired as Tier 2 support; promoted to Team Lead in June 2023. Recognized publicly by the VP of Sales & Marketing for the call and case routing overhaul.
  • Completed a NetSuite-ZOHO integration proof of concept; 2024 review: Exceeds Expectations across quality, initiative, technical skills, and dependability.
Senior Support EngineerDec 2018 - Mar 2022

SAP Concur · Bellevue, WA

  • Maintained 98% customer satisfaction across a high-volume enterprise Tier 3 queue for SAP Concur expense and travel products.
  • Served as Comms lead on the Incident Response Team for P1/P2 issues. Owned customer and stakeholder communication under SLA pressure while Tech and Lead drove resolution.
  • Queried client data with SQL to diagnose enterprise issues during Tier 3 troubleshooting; mentored team members and built performance dashboards.

Selected Projects

Architecture Zero

mrjstickel.com/projects/az

Open-source AI platform (Apache-2.0) | live production instances | published eval numbers

  • Engineered hybrid retrieval: vector search over-fetches a candidate pool, a from-scratch Okapi BM25 leg runs across the full corpus, and a cross-encoder reranks the pool down to a small kept set. Measured on a pre-registered eval, not asserted: recall lifted from 82% to 90% (tuned set) and 77% to 85% (full cohort) on a corpus that had nearly doubled, each change adopted only on a rule written before its run and every conversion verified at the chunk level.
  • Built content-addressed delta ingestion: only new or changed text embeds on updates (~10 chunks instead of ~470), a mid-ingest failure resumes cleanly, and every ingest emits an observability event.
  • Designed department-scoped knowledge bases with tiered access and a federated mesh (Eco Mode): parallel peer fan-out, per-caller scoped keys, fail-closed boundaries, graceful degradation when a peer is down.
  • Secured the fleet: OWASP Web + LLM Top 10 mapping, CI gates (gitleaks, pip-audit, bandit, trivy), guest budget caps, per-IP rate limits, injection checks, and adversarial self-audits with published postmortems.
PythonFastAPIChromaDBOllamaONNXDockerNginxGCP

Northwind AI (live demo)

northwind.mrjstickel.com

White-label client demo of Architecture Zero: click it, ask it anything

  • Live public deployment proving the white-label model: a fictional company's branded assistant with real department access boundaries. The "View as" switcher shows Sales seeing pricing docs a public visitor is refused.
  • Public trust panel (northwind.mrjstickel.com/#trust) derives every number from stored eval runs at request time: per-cohort scores never blended, corpus-fingerprinted, and the tuned-vs-holdout overfit gap published even when negative.
  • Runs as a live federated company: a hub federating over three department instances, each holding only its own documents. Per-persona routing to peers, per-peer credentials referenced by name, refusal decided after the federated merge, and per-peer citations in the answer; isolation by data placement, pinned by tests and verified live per persona.
  • Built permission-aware enterprise doors on one identity seam: Google Drive and Microsoft 365 (SharePoint + OneDrive via Graph) connectors that mirror each file's real sharing permissions onto per-document ACLs and fail closed; sign-in through Microsoft Entra or Google Workspace; a Teams bot and a Slack app that answer as the asker's own account, with only what that account may read (the Teams bot by driving the instance's own chat route, so every permission check, output filter, audit row and citation is the one the web app already has).
Architecture ZeroRAGRBACDockerGCP

Kintsugi (private)

Personal AI over a life's knowledge: the private flagship instance

  • Mobile-first (React Native/Expo) private assistant over a personal corpus: session history, dated-log chunking with recency weighting, department routing, GPU-tunneled local inference (Cloudflare tunnel to RTX 5090), and the eval discipline the case studies document.
  • Private by design. The live public counterpart to walk through is Northwind AI.
  • Published "Anatomy of a Trustworthy Answer" (mrjstickel.com/projects/anatomy-of-a-trustworthy-answer), a long-form technical paper that takes the assistant apart layer by layer (each layer's failure, the machinery built in response and its measured number), ending on a pre-registered evaluation gate cleared with its thin margins stated; every claim fact-checked against the project record before it went public.
PythonFastAPIReact NativeExpoOllamaCloudflare

Agentic HR Demo

mrjstickel.com/projects/agentic-hr-demo

Governed agentic writes | Claude tool-use agent | one trust layer behind web, Slack and MCP

  • Built a Claude tool-use agent that performs HR-system actions in plain English against a mock of Workday's documented Staffing API, behind a governance layer the model cannot bypass: authorization checked against the session caller (never model output), content-derived idempotency, field-level authorization, and refusals audited first-class. The same governed tools served to a web app, a Slack bot and an MCP server.
  • Moved the agent's write approval out of the model's reach after an outside review showed the model could approve its own staged write: a person confirms in a web approval box or on a Slack button, each plan bound to a one-time approval code, its own conversation, the person who asked and an expiry. Published the eval run before the fix (18/19) beside the run after (19/19), not just the clean number.
PythonFastAPIClaude APISlackMCPDocker

JobTrue

jobtrue.ai

AI job search platform · Solo build · Stripe billing

  • Solved AI resume hallucination structurally: the LLM selects from the user's original bullets by index and application code reassembles verbatim text. A server-side grounding layer drops and logs any fabricated item before storage.
  • Identified and remediated a critical SSRF vulnerability in the URL scrape route: blocks private IP ranges, loopback, link-local, and internal hostnames before fetch to prevent cloud credential exposure.
  • Shipped Stripe billing (checkout, idempotent webhooks, customer portal), per-user AI rate limiting via Redis sliding windows, fit-score calibration with two-shot anchoring, cost-routed models, and a full admin console with token-cost monitoring.
  • Built the application pipeline tracker: frozen resume + cover letter snapshots per submission, interview rounds, contacts, and fit-score history.
Next.jsFirebaseClaude APIStripeVercel

MySQL Client & SQL Trainer (private)

Full-stack database tool · Solo build · Mobile + API

  • Mobile MySQL client with an AI assistant that reads the actual schema and writes or explains SQL: constrained read-only, Fernet-encrypted credentials, multi-statement blocking, explicit write-mode opt-in.
  • 40-challenge SQL trainer on an isolated in-memory SQLite database, graded by automated row-set comparison.
React NativeExpoFastAPIMySQLSQLiteClaude API

TheStatic.tv

thestatic.tv

Production Streaming Platform · Solo build · 160+ API endpoints

  • Engineered GCP Livestream integration: RTMP ingest, tier-based HLS transcoding (720p-1080p60), CDN delivery, and batched VOD archival for streams of 5,400+ segments.
  • Architected the Spark Economy billing engine: Redis distributed locks (120s TTL), viewer-scaled burn rates (1x to 50x), a cron Governor deducting credits every minute, and Polygon crypto payments with verification windows.
  • Deployed the ZeroPointPass ERC-721 contract to Ethereum mainnet with EIP-2981 royalties; on-chain ownership syncs to platform tier every 12 hours.
  • Unified multi-auth in a single authorization layer: Sign-In-With-Ethereum (EIP-4361) with single-use server nonces, Decentraland signed-fetch bound to the registered scene, and Firebase custom tokens.
  • Rebuilt the payment path so the server prices every purchase and a payment is consumed exactly once, by the wallet that sent it, in the same database transaction as the benefit. Proven under true concurrency against a database emulator, with a control showing the same code without the transaction's isolation credits every racing claim.
Next.jsFirebaseGCPPolygonRedisEthers.js

What I Deliver

· Production AI systems end-to-end: RAG pipelines with measured retrieval quality (eval-stamped recall + LLM-judged answer floors), agents with tool use, guardrails, and cost observability
· Private / self-hosted AI platforms: your data, your brand, your infrastructure (Architecture Zero: live instances from one open-source core)
· Retrieval engineering: hybrid vector + BM25 fusion, cross-encoder reranking, section-aware chunking, delta ingestion. Fixed at the corpus, not patched at the prompt
· Continuous AI evaluation: question sets, regression floors, measured before/after on every change
· LLM fine-tuning through the full QLoRA lifecycle, and the senior call of when NOT to fine-tune
· Multi-provider LLM architecture: Claude, GPT, and local Ollama with failover; local inference on 24GB-class GPUs
· Federated / multi-tenant AI: department-scoped knowledge, role-based access, fail-closed boundaries, adversarially self-tested
· AI security in production: OWASP LLM Top 10 mapping, prompt-injection defense, CI security gates, published self-audit postmortems
· Documentation as infrastructure: single source of truth, generated facts that cannot drift, docs ingested into the AI itself. Measured onboarding drop from 3 weeks to 3 hours
· P1/P2 incident communication and SLA-driven operations: 13+ years of enterprise support instincts wired into everything above
· Ops resilience as machinery: immutable nightly backups, an automated daily restore drill, and fail-closed status probes wired to alerting. An untested backup is a hope
· AI-assisted engineering at scale: parallel coding agents under proof gates (a test that fails on the old code, every diff read, the full suite after each landing)

Built with JobTrue · Create your free resume →