Hyper-personal Cognitive Engine

The Path :

MoE Foundations — sparse vs. dense compute, expert networks, gating/routing, top-k, load balancing, capacity.
Routing Signal Design — what routers normally condition on, and how you'd architecturally change that to accept an external state signal.
Cognitive Load Theory — the actual cognitive-science grounding (Sweller): what "mental state" even means in a measurable sense, so "cognitive-load router" isn't just a buzzword.
Estimating Mental State Computationally — turning behavioral signals (latency, query complexity, session depth, edit patterns) into an actual feature you can feed a model.
Constitutional AI — Anthropic's approach: SL-CAI, RL-CAI/RLAIF, how a written constitution governs behavior without per-example human labels, and how that could plausibly "govern" an MoE.
Information Theory: Entropy & Compression — why "high-compression, low-variance output" is a precise, not a marketing, phrase.
Dynamic Penalty Functions — regularization, KL penalties (you know this from DPO), and designing a penalty that's a function of the load estimate rather than a fixed constant.
System Integration — wiring 1–7 into one pipeline end to end.

Phase 1: Mixture of Experts — Foundations

The problem MoE solves. In a normal transformer, every token passes through every parameter — the same FFN block, every layer, every token, no exceptions. That's dense compute: more parameters always means more compute per token. MoE breaks that coupling. The idea: have many FFN blocks ("experts") instead of one, but only send each token through a handful of them. Total parameters go up, but compute per token stays roughly flat. This is exactly why it's the right architecture for "lightweight" — you get a large effective model with a small active footprint per inference step, which matters a lot if this IAS eventually needs to run on-device.
Experts. Nothing exotic — an expert is just a normal FFN block, the same kind you already know from a transformer layer. A MoE layer just has, say, 8 of them sitting side by side instead of one.
The router (gating network). This is the new piece. It's a small learned network — usually just a single linear layer — that looks at a token's hidden representation and outputs a score for each expert:
- scores = softmax(W_router · x)
- Think of it like a hospital receptionist: the patient (token) walks in, the receptionist doesn't treat them, they just read the symptoms and send them to the right specialist (expert).
Top-k routing. You don't send a token to all 8 experts — you take the top-k scoring ones (k=1 or k=2 is common) and only run those. Everything else is masked to zero. This is the actual sparsity: for any given token, most of the network is switched off. Shazeer's 2017 paper introduced this ("sparsely-gated MoE"); Switch Transformer simplified it to k=1 for efficiency; Mixtral popularized k=2 as a sweet spot; DeepSeekMoE later split experts into many small "fine-grained" ones plus a few "shared" experts that every token always sees (a stability trick — worth remembering for Phase 8).
The load balancing problem. Here's the failure mode you need to understand deeply, because it's the single most-asked "do you actually understand MoE" interview question: if you just train router + experts jointly with no correction, the router tends to collapse — it discovers early that expert 3 is "good enough" and routes almost everything there, starving the other experts of gradient signal, which makes them worse, which makes the router trust them even less. A vicious cycle. The fix is an auxiliary load-balancing loss — an extra term added during training that penalizes the router for sending too many tokens to too few experts, pushing the token distribution across experts back toward uniform. You'll also hear "capacity factor" — each expert has a hard cap on how many tokens it'll accept per batch; tokens that overflow past that cap get dropped (skip that layer) or passed through a residual. This is a real engineering constraint, not just theory — it's what keeps training stable and batches balanced.
Why this matters for your router (a preview of Phase 2): your router is going to condition on more than semantic content — it needs a cognitive-load signal too. Load balancing becomes even trickier there, because "route by mental state" could easily correlate with time-of-day or session length in ways that create the exact same collapse problem, just driven by a different variable. Keep that in your back pocket; we'll return to it.

Phase 2 : Routing Signal Design

What the Phase 1 router actually sees. Go back to the router equation from Phase 1: scores = softmax(W_router · x) . That x is a token's hidden state — but by the time it reaches a routing layer, it's already been through several attention blocks, so it's soaked in context: meaning, syntax, position, everything upstream. So when people say MoE routes "by semantic similarity," what they really mean is: the router is a function of content only. There's no mechanism for anything outside the text — like how tired or overloaded the user currently is — to touch the routing decision at all. That's the exact gap your CV bullet is naming: "based on estimated user mental state rather than semantic similarity alone."
One honesty note worth having for an interview: empirically, MoE experts don't always specialize into clean topics the way people assume ("expert 3 = medicine, expert 7 = poetry"). Studies on Switch Transformer found a lot of specialization is by shallow patterns — punctuation, numeral tokens, syntax — not tidy semantic categories. Good to know so you don't oversell what content-based routing gives you for free.
The actual architectural question. You have a load signal (for now, just treat it as a given number or small vector — how it gets computed is Phase 4). How does it get into the routing decision? There are a few real options, each a genuine design choice:
- Option A — Concatenation. Glue the load signal onto the hidden state before the router sees it: router_input = concat(x, load_embedding) → scores = softmax(W_router · router_input) Most expressive — the router can learn arbitrary interactions between content and load. But it's also the least controllable: with joint training, there's a known failure mode where one input signal just dominates gradient and the router quietly learns to ignore the other. If load doesn't vary much in your training data relative to content, the router may learn to ignore it entirely.
- Option B — Additive bias. Keep the router untouched, but add a learned nudge conditioned on load: scores = softmax(W_router · x + b(load)) where b(load) is a small function (even a lookup table over a few load buckets) producing a per-expert bias vector. High load pushes the bias toward whichever experts you've designated as "cheap/simple." This is more interpretable — you can literally point to b(load) and say "this is the mechanism that makes overload steer routing," which matters a lot for defending a design in an interview.
- Option C — Temperature modulation. Don't touch which experts get favored — touch how decisively the router commits: scores = softmax((W_router · x) / T(load)) Lower temperature under high load → sharper, more deterministic routing (few experts dominate completely instead of a soft blend). This one doesn't change which expert wins, just how confidently — and it's a direct preview of the "low-variance" requirement in your CV bullet, which we'll fully connect in Phase 6.
- Option D — Two-stage (hierarchical) routing. First route by load tier (e.g. low/medium/high — could even be a simple threshold, not learned) to a sub-pool of experts assigned to that tier, then route by content within the sub-pool. This is structurally similar to DeepSeekMoE's shared-vs-routed split, just sliced along a different axis. It's the least flexible option, but the most guaranteed to actually work with a small amount of training data — since the load→expert-pool mapping isn't learned at all, it can't fail to matter, unlike Option A.
A load-balancing wrinkle this creates. Remember the auxiliary load-balancing loss from Phase 1, which punishes the router for over-favoring a few experts? Once load is part of routing, you want some imbalance — tokens under high cognitive load are supposed to correlate with routing toward the "cheap" experts. If you apply the vanilla load-balancing loss globally, it'll fight you and try to flatten out that correlation, which is exactly the behavior you engineered on purpose. The fix: balance within each load tier, not across the whole batch — so the loss still prevents pointless collapse (e.g., one expert hogging all "high load" tokens) without erasing the load→expert relationship you actually want.

Phase 3: Cognitive Load Theory

This phase is the cognitive-science grounding that makes "cognitive-load router" a real, defensible term instead of a phrase that sounds impressive but means nothing precise. Skip this and in an interview, the moment someone asks "okay, but what is cognitive load, technically?" you'll be stuck restating your own architecture.
The working memory bottleneck. The core empirical fact underneath all of this: working memory — the mental workspace you use to actively hold and manipulate information right now, as opposed to long-term memory where things are stored — has a hard, small capacity. Classic estimate (Miller, 1956) is "7±2 chunks"; later, more careful work (Cowan) puts it closer to 3–4 chunks. Either way, it's small, and it doesn't scale with how smart or expert someone is — an expert just packs more information per chunk (a chess grandmaster sees "a fork" as one chunk where a novice sees six separate pieces). This bottleneck is the actual thing your router is trying to protect. "Cognitive overload" isn't a vague feeling — it's this specific buffer overflowing.
Sweller's Cognitive Load Theory — three types of load. This is the part you'll cite directly:
- Intrinsic load — complexity that's inherent to the material or task itself. You can't remove it without changing the task. Understanding DPO's loss function has some irreducible intrinsic load; there's no way to make it zero effort, only more or less well-scaffolded.
- Extraneous load — complexity added by how something is presented, unrelated to the actual task demands. Bad formatting, unnecessary jargon, a cluttered UI, a rambling explanation that buries the point — all extraneous. This is pure waste; the goal is always to drive it toward zero.
- Germane load — the effortful processing that actually builds understanding: relating new information to what you already know, constructing a mental model. This is desirable effort. It's the "productive struggle" — working memory being spent on learning, not wasted.
Why this maps directly onto your own project's philosophy. You've already got a founding principle at Xlon Labs called Friction-Preservation — that removing all friction starves the mental effort that builds capability. In this vocabulary, that's precisely: minimize extraneous load, but don't touch germane load. This actually creates a sharp design constraint for the penalty function you'll build in Phase 7 — if the dynamic penalty just crudely compresses output whenever it detects "overload," and it can't distinguish extraneous load (the user is confused because of bad phrasing) from germane load (the user is working hard on something appropriately difficult and that's good), it'll flatten both. That would mean your overload-detector actively undermines your own mission — it'd simplify away exactly the productive difficulty the IAS is supposed to preserve. Worth sitting with that tension now, because it's the single best interview question you could get asked about this project, and it's also a real open design problem you'll have to make a call on in Phase 4/7.
How load is normally measured (and why most of it isn't available to you). In the HCI/cognitive-science literature, load is measured a few ways:
- Subjective — self-report scales like NASA-TLX, filled out after a task. Useless for real-time routing; nobody's filling out a survey mid-conversation.
- Performance-based — dual-task methodology: give someone a secondary task and measure how much it degrades under load. Not practical for a conversational assistant.
- Physiological — pupil dilation, EEG, galvanic skin response. Real, sensitive measures — but they need dedicated sensors your IAS almost certainly won't have.
None of the "gold standard" measurement methods are available to a text-based assistant. Which means — and this is the honest, slightly uncomfortable truth you should walk into Phase 4 with — you're not going to measure cognitive load in the rigorous experimental sense. You're going to infer a proxy for it from behavioral traces (response latency, message complexity, edit/backtrack patterns, session length). That's a legitimate and common approach in applied HCI work, but it's a proxy, not ground truth, and you should describe it that way rather than overclaiming.
One honesty caveat for later: Sweller himself revised the theory over the years, and in some later versions germane load gets folded into intrinsic load rather than treated as fully separate — the three-way split isn't universally settled science, it's the most commonly taught version. Fine to use it as your working framework (it's the standard reference), just don't present it as unimpeachable if pushed.

Phase 4: Estimating Mental State Computationally

The honest framing first. You're not detecting cognitive load — you're building a heuristic proxy from behavioral traces and treating it as if it correlates with load. That's a legitimate, standard move in applied ML (same logic as using click-through rate as a proxy for "interest"), but keep the language in your head precise: proxy, not measurement.

Cheap, available signals — pick from these, don't invent exotic ones. Everything here is something you already have in a normal text conversation, no new instrumentation needed: [The response has follow up questions asked most of the time]

Response latency — time between your output and the user's next message. Very short = fast/confident; unusually long = either deep thought (fine) or stuck (not fine) — ambiguous alone.
Message length trend — shrinking messages over a session can mean fatigue; long, elaborate messages can mean either engagement or frustrated over-explaining.
Query complexity — rough proxies: sentence length, number of embedded sub-questions, vocabulary rarity. Cheap to compute (even just token count + a syntax parse) without needing a separate model.
Backtrack/repair rate — how often the user rephrases, says "no I meant," or re-asks something close to a prior message. Strong signal — repair almost always means the last output didn't land.
Session depth — how many turns into a continuous session you are. Load tends to accumulate across a session even if no single message looks hard.

Now, splitting these into extraneous vs. germane — the lean version. You don't need two separate models. The cheapest version that still respects the distinction: use the same signals, but read them against whether your own last output was the likely cause.

If backtrack/repair follows right after your response — that's extraneous load. The confusion was caused by how you said something, not by the task's inherent difficulty. Signal: repair_rate conditioned on turns_since_last_assistant_message == 1 .
If latency and message complexity are climbing gradually across a session, with no repair events, and the user's own messages show they're making incremental progress (not just repeating themselves) — that's more likely germane load. They're working, not stuck.

That single conditioning trick — is confusion immediately downstream of my last output, or is it accumulating independent of it — is doing almost all the work of the distinction, for close to zero extra engineering. It won't be perfect, but it's honest and cheap, which is exactly what "less time and energy" calls for.

Turning it into a number. Simplest workable version: compute each signal, normalize to [0,1] against a rolling session baseline (not a fixed global threshold — everyone's "normal" typing speed and message length differs), then combine as a weighted sum into two scalars: extraneous_load and germane_load . Weights can just be manually set for a first version — you don't need to learn them, and hand-set weights are much easier to explain and defend than a trained sub-model you can't fully interpret yet.

extraneous_load = w1·repair_rate_after_response + w2·latency_spike
germane_load    = w3·session_depth_norm + w4·complexity_trend

This feeds Options B/D from Phase 2 as the load input, and - as flagged in Phase 3 - a real version 2 would only let extraneous_load trigger simplification, since suppressing germane_load is exactly what Friction-Preservation says not to do.

Phase 5: Constitutional AI

Why not just standard RLHF? Quick recap of the problem RLHF solves and where it strains: you collect human preference labels (A vs. B, which is better), train a reward model on those labels, then optimize the policy against that reward model via RL. Three real cracks in this at scale: it needs a lot of human labels, which is slow and expensive; human labelers have to actually read harmful/toxic content to label it safely, which is a genuine cost to the people doing the labeling; and the "values" being optimized for are never written down anywhere — they're implicit, scattered across thousands of individual crowdworker judgment calls, which makes the resulting behavior hard to audit or explain. Anthropic's Constitutional AI paper (Bai et al., 2022) is a direct response to that last point specifically: what if the values were explicit and legible instead of implicit?

The two stages.

Stage 1 — SL-CAI (Supervised Learning). Start with a helpful-only model (no safety training yet). Feed it prompts designed to elicit bad behavior. Get its (probably bad) response. Then — and this is the actual trick — show the model its own response alongside a principle pulled from a written constitution (a list of plain-language rules, e.g. "choose the response that is least likely to be harmful," "choose the response that is more honest"), and ask the model to critique its own output against that principle, then revise it. Repeat critique→revise a few rounds. Fine-tune the original model on these self-revised outputs via ordinary supervised learning. No human ever labeled anything here — the model is bootstrapping its own better behavior off a written document.
Stage 2 — RL-CAI (a.k.a. RLAIF). Same shape as RLHF's RL stage, but instead of humans producing the preference labels for the reward model, the model itself compares pairs of its own outputs against the constitution and generates the preference labels ("AI feedback" instead of "human feedback"). Train a reward model on those AI-generated preferences, then RL against it, same as before.

The one-line version: RLHF encodes values implicitly through thousands of human judgments; CAI encodes them explicitly through a document the model consults, criticizes against, and revises toward. That's the entire conceptual shift — everything else is standard SL/RL machinery you already know the shape of.

A connection you should notice immediately: later work (including Anthropic's own follow-ups) showed the RL/PPO step in RL-CAI can be swapped for DPO directly on the AI-generated preference pairs — same constitutional self-critique process to generate the pairs, but you optimize with DPO instead of running a full PPO loop. You already built a QLoRA + SFT + DPO pipeline for your alignment project. That pipeline doesn't change here — only where the preference pairs come from changes: instead of your ~3k human/curated cognitive-behavioral pairs, you'd generate pairs by having the model critique-and-revise its own outputs against a constitution. Same training code, different data source. That's genuinely reusable infrastructure, not just a conceptual parallel.
- PPO : Proximal Policy Optimization : A popular RL algorithm that acts as a stable, efficient method for updating an agent's behavior policy without causing destructive, overly large performance drops.
How this plausibly attaches to your MoE (the "governed by a constitutional framework" part). Given the lean-scope instinct you've had all along, full RL-CAI training (reward model + RL loop) is a lot of infra for a CV-scope project. The honest, defensible, lightweight version: implement the critique-revise loop at inference time, not baked into weights via full RL training. Concretely — after the MoE produces an output, run one more pass: "here's the constitution, here's what you just said, does it violate any principle? If so, revise." That's SL-CAI's core mechanism, applied live as a governance layer sitting over the router+experts, rather than trained into them. It's cheap, it's demoable, and it's an accurate, non-oversold description of what you built: "implemented the constitutional self-critique/revision mechanism as an inference-time governance layer, rather than the full RL-CAI training pipeline." That sentence survives an interview follow-up.
One more thing worth noticing, specific to you: the constitution isn't some abstract document you need to invent from scratch — Anthropic's actual constitution draws on sources like the UN Declaration of Human Rights plus its own principles. You already have your own source material: Cognitive Sovereignty and Friction-Preservation are exactly the shape of a constitutional principle ("prefer the response that builds the user's own capability over one that just hands them the answer"). Your project's constitution could quite literally just be Xlon Labs' founding principles, formalized into rule form. That's not a coincidence you have to manufacture — it's already sitting in your own design philosophy.