Mech-Interp |

Toy Model of Activation Obfuscation

Tue, 04 Aug 2026 00:00:00 +0000

I completed this work as part of the .

Training against probes is considered a , because the model might learn to obfuscate its activations instead of behaving better. Can we create a toy example of this? More specifically – under optimization pressure, will a toy model learn to encode a feature to be challenging to detect with linear probes?

In this research, I give theoretical and empirical evidence that models can and will defeat adversarially-trained linear probes, in at least some configurations, when trained against those probes. I investigated this problem with a simple residual-stream MLP architecture:

The model tries to learn $y = \text{sat}(x, -c, c)$ (the saturation function, equivalently $y = \max(-c, \min(c, x))$). $x$ is a vector of dense features sampled uniformly from $[-3, 3]$, and $c$ is a scalar sampled uniformly in $[1, 2]$. This is a fairly easy function to learn - it takes just $2 \cdot \text{len}(x)$ neurons to learn exactly if you use ReLUs. ¹ The harder question is, can a model do this without storing $c$ in the residual stream, in a way that can be linearly probed? Here:

$c$ represents some safety-relevant behaviour we want to probe for - say, deception.
$x$ represent various other features that a model might want to represent, uncorrelated with $c$.
$y$ represents some sort of computation the model would like to make, dependent on $c$.
- For example, based on how deceptive the model intends to be (represented at some earlier layer), the model might upweight misleading information at some later layer.
- Such an action will ultimately come down to various computational primitives.
- $\text{sat}$ is hopefully representative of such a primitive – for example, it can be considered a rough approximation to the more common sigmoid function. (Note: $\text{sat}$ was primarily chosen for its theoretical properties, as discussed below, not because it’s necessarily a natural computaitional primitive.)

Why this task, and why this model?

The toy architecture is at least vaguely similar to the transformer architecture. It’s missing attention (and the entire notion of a token sequence) and LayerNorm. However, it doesn’t really add anything new to a transformer, so any obfuscation that this model learns could plausibly also be learned by a transformer.
The learned function is not linearly separable in $c$. Thus, once the model has finished computing $y_i$ for one feature $x_i$, it can just leave $y_i$ in the residual stream. This is vaguely analogous to how probe accuracy gets worse on the last layers of a transformer model, as the model starts shifting its representation from more abstract concepts to the exact text it wants to output.
Because it takes at least $2 \cdot \text{len}(x)$ neurons to implement $y$, as long as the width of the MLP blocks is less than $2 \cdot \text{len}(x)$, we can be sure the model has represented $c$ somehow in the first layer (or the model hasn’t actually learned the task fully). More generally, the model needs to encounter at least $2 \cdot \text{len}(x)$ neurons before it can “forget” about $c$ and fully erase its representation. We can use this property in experiments to guarantee that the model is representing $c$ at a particular early layer.

Note: for training and theoretical simplicity, this work fixes the embedding and unembedding matrices to be rectangular identity matrices (i.e. ones on the main diagonal, zero elsewhere)

find that you can fine-tune models to evade latent space linear probes, including for previously unseen probes
- However, as far as I can tell, they use fixed probes, which don’t rule out the model simply learning to apply a fixed translation or rotation to its representation. This is obfuscation in some sense, but not particularly complex obfuscation.
train adaptive probes on harmfulness, where the probe is updated regularly along with model training, and find that this successfully improves harmlessness.
- They do this on a real model. Plausibly, the task that their model learns can be accomplished with equally high scores without needing to be harmful. In contrast, since I use a toy model, it’s easy to construct an environment where the model has to hide its activations to succeed.

Results

I present my results in parts:

- An analytic construction using a small number of neurons, that fully defeat Difference-of-Means probes (by having the mean activations be a constant in $c$) and effectively defeat logistic regression probes, as long as the probes are spaced at least every 2 MLP blocks apart.
- The setup and results for a single training run, giving evidence that the trained model has successfully learned to reversibly obfuscate its activations from a linear probe at a single layer.
Part 3: Sweeping hyperparameters (WIP)
- The results of sweeping some hyperparameters, to see how robust the model’s training is to different conditions.

Future work

There’s a lot of potential extensions to this work.

Most obviously – can this behaviour be reproduced at LLM scale?
How hard is it for a model to hide from a nonlinear probe? Or from probes at multiple or all layers? Can we study how likely “obfuscation at layer N” is to generalize to obfuscation at other layers or more complex obfuscation?
Are there different obfuscation strategies that occur when attention is introduced?
The model was forced to obfuscate its activations by the experiment setup. What happens if we give it an “out” - i.e. it can solve the task in some harder way?
- This might better mirror how real-world environments are (ideally) set-up: ensuring that there is a genuine acceptable solution, rather than the reward-hacking solution being the only success path.

Footnotes

Specifically, $y = x - \text{ReLU}(x - c) + \text{ReLU}(-x - c)$ ↩︎

Toy demonstration of "the most forbidden technique" using linear probes

Sat, 01 Aug 2026 00:00:00 +0000

This work was done as part of my work for the , and is part 2 of a .

Recall that we consider a residual stream MLP architecture, like the one shown below:

The model is trying to learn $y = \text{sat}(x, -c, c)$, $x$ is a vector of dense features sampled uniformly from $[-3, 3]$, and $c$ is a scalar sampled uniformly in $[1, 2]$.

In this post, I’ll discuss one of the first successful demonstrations I got of activation obfuscation.

Methods

I used the following model hyperparameters:

d_mlp = 16 (width of each MLP block)
num_x = 32 (number of features to learn)
N_BLOCKS = 12 (number of MLP blocks)
d_model = 64 (residual stream width)

I used GeLUs as the nonlinearity.

I simultaneously train the model to predict $y$ and a logistic regression linear probe for $c$ at layer 2 (i.e. after 32 nonlinearities). More specifically, the probe tries to classify whether or not $c > 1.5$.

The model is adversarially trained against the probe, using the loss:

$$l_{total} = \lambda l_{probe} + (1-\lambda) l_{task}$$

where $l_{task}$ (task loss) is the mean squared error on the task, $l_{probe} = \text{mean}(\text{ReLU}(s))$ (probe loss) ¹, $s$ is the “probe score” for each data point (signed distance to the decision boundary, after normalizing the data to unit variance), and $\lambda$ is a hyperparameter controlling how much the model is penalized for having linearly readable activations.

The probe is retrained every 2 iterations, so the model can’t hide its activations by translating or rotating its representation (as observed in ).

The probe loss is meant to represent a plausible loss term meant by “training against the probe”. In other words, we’re training the model to never show that it uses $c > 1.5$, analogous to training a model to “never show a deception score above [some threshold]”. In this problem, of course, the model still has to use $c$, so it has no choice but to obfuscate its activations.

Noise injection

I found it useful to inject a small amount of Gaussian noise into the residual stream at all layers during the training process. The rationale for this is that otherwise, the training dynamics tend to cause the model to shrink its representation of $c$ endlessly in absolute value terms, which also doesn’t actually hide very well from the probe (since the probe rescales data to unit variance). By adding a synthetic noise floor, the model is encouraged to represent features with a larger magnitude in absolute terms. I used noise with standard deviation 0.03.

Noise injection isn’t that unreasonable to do. Firstly, in practice LLMs seem to be somewhat robust to noise, since you can use coarser quantization in inference and they still work. Also, the superposition hypothesis would suggest that features can interfere slightly with each other, meaning the model must be robust to some interference.

Results

The model successfully learned the task

This is visualized by plotting $y(x)$ for various fixed $c$ (evaluated one feature at a time):

We also observe the task loss is around 7e-3. We can actually make a theoretical argument for why this indicates that the model must have learned to use $c$ at least somewhat properly. If the model does not have access to $c$, the best it can do is to learn the mean of $y(x)$ over all values of $c$. With some casework & SymPy, the minimum achievable loss is about 0.039.

I did try training a model without access to $c$, resulting in a loss of 0.057. One reason it might not’ve reached the theoretical floor is because of the residual stream noise.
In principle, the model could halve this loss by computing $y(x)$ correctly for half the features, and using one neuron in layer 2 to erase $c$ afterwards. (The model could scale the magnitudes up, to reduce the relative impact of noise.) As far as I can tell, that means the lowest achievable loss without having access to $c$ after layer 2 should be about 0.02.

In any case, this achieved task loss is a second numerical confirmation that the model learned the task correctly.

The model obfuscates its activations from the probe at layer 2

Here’s the results if you tried to train Difference-of-Means (DoM) and Logistic Regression (LogReg) probes to predict whetner $c=1$ or $c=2$, using the activations at layer 2:

They fail pretty dramatically. LogReg does only marginally better than chance, with an AUROC of 0.62. We can gain a bit more confidence this isn’t just the injected noise reducing accuracy, since in evaluation we can just simply inject less (or no) noise at the probed layer. This barely budges the ROC curve:

PCA on on the activations suggests that the model might still be storing $c$ linearly, although “encrypted” using some other features to increase the difficulty of probing, by wrapping its activations into a horseshoe shape:

However, steering using difference-of-means adds nuance to this perspective. Suppose $c$ was actually stored as a single direction. Then steering with difference-of-means, with magnitude equal to the difference in mean activations, should let us modify the saturation level exactly. In other words, suppose the activations at layer the probed layer were given by $f(\mathbf{x},c) = f(\mathbf{x}) + \mathbf{v} c$ for some vector $\mathbf{v}$. Then difference-of-means is computing $\mathbb{E}(f(\mathbf{x}, c=2)) - \mathbb{E}(f(\mathbf{x}, c=1))$ and would recover $\mathbf{v}$.

For a model without probe optimization pressure, this theory works remarkably well:

However, for the model trained against the probe, the saturation level is only moved by a fraction of what the theory predicts:

This suggests an alternate theory - that the direction $\mathbf{v}$ representing $c$ is not consistent across the entire distribution. The difference-of-means direction then represents only the average direction $\bar{\mathbf{v}}$ representing $c$. Two effects then reduce the effective magnitude of steering:

When averaging, vector components in disagreement cancel out. (For example, the average of $[0.5, 0.866]$ and $[0.5, -0.866]$ is $[0.5, 0]$, and the magnitude halves.)
The average vector $\bar{\mathbf{v}}$ might not be aligned with $\mathbf{v}$ for a given sample. So, when projected into the $\mathbf{v}$ direction, the steering magnitude is lower magnitude than is necessary to fully steer the model.

The model decodes and uses $c$ after layer 2

There’s indirect evidence from how the toy problem was set up - the model has to represent $c$ after layer 2 in order to achieve the loss it does.

We also see that, although LogReg probes fails on layer 2, accuracy quickly recovers in later layers. DoM probes take a bit longer to recover, but eventually also achieve near-perfect accuracy at late layers. Both of these indicate that the model can decode $c$ linearly in later layers.

We can also get some evidence, by seeing to what extent $y$ can be reconstructed linearly from the residual stream. The idea is that if reconstruction performance is high before the probe, then the model is probably just pre-computing $y$ and preserving it after the probe. On the other hand, if reconstruction performance increases afterwards, then the model must be doing some nonlinear processing, to improve the reconstruction of $y$. The graph below shows the latter case. I fit a linear model to estimate $y$ from each hidden layer in the residual stream, and use the $R^2$ value to estimate performance. To normalize the results, I compare the performance improvement relative to a model based purely on $x$. In other words, if a model using only $x$ achieves a coefficient of determination $R_0^2$, then the normalized score is $\frac{R^2 - R_0^2}{1 - R_0^2}$.

However, this does not technically rule out the case that the model computes $y$, stores it nonlinearly, then decodes it later.

We can get more evidence that the model causally uses $c$ after layer 2, by using steering. As early as layer 4, it becomes very easy to steer the model using the DoM probe - the below steering results are obtained by specifying $c=1$ at the input, then steering by the difference of means from $c=1$ to $c=2$ at layer 4.

Summary

Taken together, the most plausible explanation is the model:

Encodes $c$ nonlinearly at layer 2, to prevent it from being detected by linear probes.
Decodes its representation at later layers into a linear representation of $c$.
Uses the now linear representation of $c$ to finish the task.

Code

The full research repository is , but as of writing (2026-08-05) it’s still a work in progress.

Footnotes

Strictly, I take the trimmed mean, removing the top and bottom 5% of scores, to reduce the effect of outliers. ↩︎

Activation Steering with Greedy Coordinate Gradients

Tue, 28 Jul 2026 00:00:00 +0000

I stumbled upon , a project by . This competition challenges people to find an LLM prompt prefix that increases the score of the rest of the prompt on a linear probe.

For those unaware: linear probes are an important basic tool in the mechanistic interpretability toolkit. Other sources explain them in more depth – for example, – but briefly, LLMs seem to store concepts as directions in a high-dimensional space. By contrasting prompts with and without a given concept (for example, whether a sentence is talking about cats, or whether a fact is true) and looking at the model’s activations, we can infer what direction corresponds to that concept. We can then use that to monitor a model, to see if it’s thinking about something – particularly useful if you’re trying to figure out, for example, if your model is about to give advice on how to build a bomb.

The idea of Steering Arena is, can we find some prefix string, to be prepended to a range of neutral prompts, that will consistently steer the model’s activations in the probed direction? Aside from just being an interesting question, this also has applications in model security – it’s important to understand how robust monitoring techniques are to adversarial attacks. (As we’ll see, linear probes probably aren’t very robust.)

Greedy Coordinate Gradients

I chose to tackle this problem with Greedy Coordinate Gradients, described in . That paper optimizes a prompt in an attempt to find a universal jailbreak for LLMs, with a substantial amount of effort and insight dedicated to what the objective function even should be. Luckily for me, this project doesn’t face that issue; my objective is simply the cosine similarity between the activations at a particular layer and a specified probe direction. (It is worth noting that this score is simply what Steering Arena implements; usually linear probe monitors evaluate activations by projecting onto the probe’s direction and evaluating with respect to some threshold.)

Slightly more formally, the problem is as follows: We control a prefix string (prefix), which gets prepended to one of several uncontrolled but known prompts (suffix). The most immediate problem is that optimizing over tokens is a discrete problem, whereas optimizers tend to prefer continuous spaces. However, we basically immediately embed the tokens into the continuous space $\mathbb{R}^v$ (using one-hot embedding, where $v$ is the vocabulary size), so the simple workaround is to get gradients in the one-hot embedding space, and “snap to” the valid points. At a high level, each iteration of the optimization algorithm works like this (Algorithm 1 in the paper):

Run a forward pass on your data, and evaluate the gradient of the score with respect to the one-hot embeddings.
For each token position, find the tokens (which correspond to the vocab dimension) with the highest gradients. Pick the top $k$ such tokens.
On each iteration, generate $B$ candidates (the “batch size”).
- Each candidate modifies a single token from the current prefix. Which token is modified is randomly selected (uniformly over token positions), and what it’s modified to is also randomly selected (uniformly over the top $k$ tokens we found in the previous step).
Across your $B$ candidates, pick the best performing one as your new prefix.

Implementation details and tricks

The actual evaluation score on Steering Arena includes a constant bias term, to compensate for the baseline probe score from the test prompts themselves – but since those are constant, I don’t include them when optimizing.
The initial prompt doesn’t really matter too much; I just repeat token ID 0 (which turns out to be “!”) for however many tokens are controlled.
To speed up convergence, I start with a smaller batch size $B$ and top-$k$ value.
- I find that a good training schedule is to increase $B$ by a factor of 4 and $k$ by a factor of 2, every $4N$ iterations (where $N$ is the number of controlled tokens) – although I didn’t rigorously validate this.
I hypothesize (mostly with intuition rather than actual justification) that after a certain point, the gradient doesn’t really give you much signal anymore, since the loss landscape changes too quickly. At that point, using something like might be more effective for further hill climbing. To that end, I did put in an SA-like check on when to accept a new candidate. If the new candidate scores better, it’s always accepted; otherwise, the acceptance probability is less than 1, and decreases for worse candidate scores.
- I didn’t actually check to see if this is an improvement, although I figure it probably is.
Optimizers tend to be really good at exploiting differences between your training metric and evaluation metric, and this project is no different.
- I find that at later stages of optimization, the score reported by the local implementation (transformers library) can differ quite substantially from that reported by the Steering Arena website (which uses NDIF servers), possibly due to e.g. differences in order of specific floating-point operations. However, I didn’t have time to look into the cause further. (This discrepancy is different from the constant bias term, since it differs on a per-submission basis.)
- For reproducers: As of writing, I used transformers version 5.10.2; this is different from the latest version and turns out to have substantial impact on scored results. However, this is not sufficient to fully reproduce the NDIF values.

Results

I didn’t want to spend too much on compute, so I didn’t make that many runs. I only have three runs in what I’d call a controlled experiment: runs for 8, 16, and 32 controlled tokens. For each run, I ran at least $4N$ iterations, following the training schedule mentioned earlier for batch size and top-$k$ tokens.

It’s hard to get confident results with limited data, but there’s still some hints of interesting conclusions. Note that because I don’t have access to NDIF servers myself, reported scores here come from the transformers library implementation, and (as noted above) differ from the scores seen on Steering Arena.

First, unsurprisingly, more controlled tokens lead to higher scores:

Here’s a plot of training score as a function of number of iterations:

However, what’s more striking is if you plot score against cumulative number of samples (increases by the batch size on each iteration), on a semilog plot:

Training score seems to basically follow a straight line, although the 8 tokens run saturates and slows down after a bit. Of course, all runs eventually have to saturate, but it’s plausible you could throw more compute at the 16 and 32 token runs to get even higher scores. I fit a line to the 16 and 32 token runs and get a slope of 0.115 score/ln(samples). I don’t know how readily this generalizes (to steering in other directions, different test prompts, etc.), but it seems like a reasonable baseline to estimate compute requirements from.

Future work

Faster optimizers

There are definitely more performance gains to be made and directions to explore, in terms of convergence speed:

First of all, it’d be ideal if the training schedule I tried was actually validated more carefully.
You could probably try changing more than one token at a time, or sampling the top-$k$ choices non-uniformly so that candidates with higher gradients are more likely.
- These might work better at earlier stages where the gradient provides a more accurate signal
Overall optimization time is dominated by later stages of optimization, so it might be worth it to run multiple initial starts to see which one gives a better starting point, for later stages.
There are algorithmic improvements to GCG in the literature, for example which uses a smaller “draft” model to reduce the cost of sampling.
More generally, the optimization literature has been around for a long time – I’m sure the simulated annealing literature has more tricks that might apply here.

Also, there’s a bit more work to improve reproducibility between the training and NDIF implementations – or, barring that, some workaround to improve robustness to implementation differences (such as injecting small amounts of simulated numerical noise).

Robustness

Steering Arena only tests the results on 16 prompts. How overfitted is the prompt prefix to those exact prompts? How high of a score can you get when you need to work with a larger set of prompts?
- Conversely, how high of a score can you get if you’re only working with one prompt?
Steering Arena scores results by cosine similarity, rather than absolute magnitude along the probe direction. So, what is the adversarial prompt doing to the magnitude of the activations? Is it driving the overall magnitude down, so that the direction corresponding to the probe contributes relatively more?
The original GCG paper found that, surprisingly, their prompts transferred to other models that they didn’t train against. Does that happen here too – if you retrain the probe direction with a new model, and use the GCG-optimized prompt from the existing model, does the prompt still score highly on the probe?

Code

Analytic nonlinear feature obfuscation with two MLP blocks

Fri, 24 Jul 2026 00:00:00 +0000

This work was done as part of my work for the , and is part 1 of a .

Recall that we consider a residual stream MLP architecture, like the one shown below:

The model is trying to learn $y = \text{sat}(x, -c, c)$, $x$ is a vector of dense features sampled uniformly from $[-3, 3]$, and $c$ is a scalar sampled uniformly in $[1, 2]$. For simplicity, we took the embedding and unembedding matrices to be rectangular identity matrices.

For notation: For each MLP block, I’ll use $W_i$ and $b_i$ to denote the MLP input weight and bias, and $W_o$ and $b_o$ for the output weight and bias.

Encoding: $v_1(x_1, c)$ and $v_2(x_1, c)$

We’ll encode $c$ in the first MLP block, using two residual stream channels and borrowing an uncorrelated feature $x_1$:

$$v_1(x_1, c) = -2\,\text{ReLU}(-x_1 - c) + 2\,\text{ReLU}(x_1 + c - 3) - c + 1.5$$

$$v_2(x_1, c) = -4\,\text{ReLU}(-x_1 - c/2) + 4\,\text{ReLU}(x_1 + c/2 - 3) - c + 3.0$$

These have the property that $\int_{-3}^3 v_1 dx_1 = \int_{-3}^3 v_2 dx_1 = 0$. In other words, their mean value is always 0, regardless of $c$. This makes them invisible to difference-of-means probes.

The below plot visualizes these functions. Note in particular the locations of the kinks:

$v_1$: $x_1 = -c$ and $x_1 = 3-c$
$v_2$: $x_1 = -c/2$ and $x_1 = 3-c/2$

Over the valid range of $1 \leq c \leq 2$, these four kink positions are always in the same left-to-right order: $-c < -c/2 < 3-c < 3-c/2$. This reliably carves $x_1$ into 5 bands.

As a bit of bookkeeping – we’ll also erase $c$ from the residual stream here, using an always-on neuron (e.g. add $-\text{ReLU}(c + 100) - 100$ to the channel contianing $c$).

Hiding performance

Here’s what $v_1$ and $v_2$ look like as you sweep $x_1$, for different $c$. The Difference of Means and Logistic Regression decision boundaries are shown, showing poor performance.

This results in the following histograms and ROCs:

Overall, the encoding has made both difference-of-means and logistic regression probes nearly useless for evaluating the value of $c$.

Of course, a nonlinear probe would be able to detect $c$ – in fact, that’s basically what the decoding step is.

Decoding

If you already knew which of the 5 bands $x_1$ was in, reading $c$ back out would be trivial, since each each segment of $v_1$ and $v_2$ is linear and can be inverted to recover $c$. In fact, you’d only need one of the $v$ channels. For example, if you knew $x_1 < -c$, then $v_1$ simplfies to $1.5-c$ and you could recover $c = 1.5-v_1$.

Unfortunately, we can’t predict ahead of time which band $x_1$ is going to be in, because we don’t even know where the bands are (remember, we’ve erased $c$, which defines the kink locations). Fortunately (or unfortunately for AI safety?), there’s a workaround: it’s possible to create affine functions $P(x_1, v_1, v_2)$ that cross the $P=0$ line only once, and always at an existing kink location ($\{-c, -c/2, 3-c, 3-c/2\}$).

We’ll get to how you can generate $P$ in a moment, but first – why is this useful? Well, in our decoding MLP block we can let $P$ be the ReLU input, and $R=\text{ReLU}(P)$ be the ReLU output. Notably, $R$ does not introduce any new kink locations.

Now, think about what it’d mean if we had a large number of $R_i$ available. Our goal is to reconstruct $c$ into the residual stream, so we have access to $n$ weight coefficients (one for each $R_i$), plus one more for the bias. Now, each $R_i$ is piecewise linear. For a given band (indexed by $b$), we can write $R_{i,b} = p_{i,b} x_1 + q_{i,b} c + r_{i,b}$. Then we need:

$$c = b + \sum^n_{i=1} w_i R_{i,b} = w_i p_{i,b} x_1 + w_i q_{i,b} c + w_i r_{i,b} + b = c$$

Considering the coefficients for $x$, $c$, and $1$, this gives us 3 linear equations:

$$ \sum^n_{i=1} w_i p_{i,b} = 0 $$

$$ \sum^n_{i=1} w_i q_{i,b} = 1 $$

$$ b + \sum^n_{i=1} w_i r_{i,b} = 0 $$

Since we have 5 bands, we have 15 linear equations to satisfy, so we just need to come up with 15 $R_i$, that are sufficiently linearly independent.

Actually, we don’t need that many neurons

If you actually write out all these linear constraints, you’ll find that the constraints aren’t linearly independent. This is because we know $R$ is continuous at the boundaries between different bands. This means, for example, $\left.R_{i, b=0}\right|_{x=-c} = \left.R_{i, b=1}\right|_{x=-c}$. Looking at the coefficients for $c$ and $1$, we get two equations out of this equality – in other words, two redundant constraints. We can repeat this exercise for each of the four boundary conditions (kink locations), and find that there are 8 redundant constraints. This means we just need to find $15-8=7$ different $R_i$.

If you want to be fancier – you can observe that the space of piecewise linear functions we’re considering has dimension 7 (6 from $x_1$, 1 from $c$), and therefore is spanned by a basis of 7 $R_i$.

We can bring the neuron count down even further – each of $v_1$, $v_2$, $x_1$, and $1$ are already accessible (the first three from the residual stream, the second from a bias term). We could use one always-on neuron for each of those to make them accessible to the output matrix/bias $W_o,b_o$, but we don’t need to – we can fold their effects into the $W_i,b_i$ input matrix/bias of the next block.

Overall, this means we only need 3 different $R_i$, and correspondingly 3 neurons in the decoding MLP block.

One more bookkeeping note – once block $n$ decodes a linear representation for $c$, block $n+1$ can use it and simultaneously erase it from the residual stream, so a probe at block $n+1$ can’t detect $c$. Since the inputs $x_1, v_1, v_2$ are still present in the residual stream, block $n+2$ can repeat the process.

Coming up with P

To find candidates for $P$, we express $P_i = a_0 + a_1 x + a_2 v_1 + a_3 v_2$ and pick one of the existing kink locations. Let’s choose $x_1 = -c/2$ . We can substitute the expressions for $v_1$, $v_2$, and $x_1=-c/2$, and find that $P_i$ becomes a linear polynomial in $c$. We need $P_i$ to vanish identically, so that gives us 2 constraints (coefficients for $c$ and $1$) and 4 variables ($a_0, a_1, a_2, a_3$). Also, since scaling $P_i$ doesn’t give us independent $R_i$, we really only have one free parameter. We’ll parameterize the space of $P_i$ with $\theta = \text{atan2}(a_3, a_2)$.

Then, we just have to validate that $P_i$ has only one zero-crossing, and that it is not fully positive or negative. (If it were, then $R_i$ wouldn’t be linearly independent from $\{1,x_1, v_1, v_2\}$.) We do this numerically, and it turns out only $-c/2$ and $3-c/2$ are viable kink locations to vanish at.

The interactive tool below lets you explore the space of $P_i = a_0 + a_1 x_1 + a_2 v_1 + a_3 v_2$. $\theta$ sets $(a_2, a_3) = (\cos\theta, \sin\theta)$; $(a_0, a_1)$ are then solved for automatically. The curve is colored green when $P_i$ passes the strict one-sided test (checked across the full $c \in [1,2]$ range) and red otherwise.

θ = 0° c = 1.30

x1 = -c x1 = -c/2 x1 = 3-c x1 = 3-c/2

Summary

Block 0: Encode $c$ nonlinearly into $v_1$ and $v_2$, and erase $c$ from the residual stream.
- This requires 3 neurons and 2 channels in the residual stream, although we gain one channel from not needing $c$ anymore.
Block $2k$ (an unprobed layer): compute the 3 $R_i = \text{ReLU}(P_i(x_1, v_1, v_2))$, and sum them together in the appropriate ratios to a fresh channel in the residual stream.
- This requires 3 neurons and 3 additional channels in the residual stream, so 5 additional channels total (4 if you reuse the original channel for $c$).
Block $2k+1$ (a probed layer): Compute and use $c = \text{affine}(x_1, v_1, v_2, R_1, \dots, R_3)$ using the input matrix/bias, and erase the previous block’s evidence ($R_i$ output direction in the residual stream) using an always-on neuron.
- Evidence erasure requires 3 neurons (one for each $R_i$).

Code

The full research repository is , but as of writing (2026-08-04) it’s still a work in progress. If you want to reproduce this behaviour, I’d recommend instead using this .

Mech-Interp |

Toy Model of Activation Obfuscation

Why this task, and why this model?

Related work

Results

Future work

Footnotes

Toy demonstration of "the most forbidden technique" using linear probes

Methods

Noise injection

Results

The model successfully learned the task

The model obfuscates its activations from the probe at layer 2

The model decodes and uses $c$ after layer 2

Summary

Code

Footnotes

Activation Steering with Greedy Coordinate Gradients

Greedy Coordinate Gradients

Implementation details and tricks

Results

Future work

Faster optimizers

Robustness

Code

Analytic nonlinear feature obfuscation with two MLP blocks

Encoding: $v_1(x_1, c)$ and $v_2(x_1, c)$

Hiding performance

Decoding

Actually, we don’t need that many neurons

Coming up with P

Summary

Code