@BlancheMinerva
@NeelNanda5 @OrionJohnston I guess it depends on where / how you think about adding the additional neurons. I was imagining adding a neuron as adding 2 d_model parameters. In some regimes that comes out quadratic but in others (e.g., adding an additional layer to the model) it doesn’t.