The Major Noticeable Simulacra Difference

I have long been skeptical when people suggest simulations somehow inherently differ from a process. They are bound to be literally different in the real world (e.g., see the no cloning theorem), but I think these differences often have nothing to do with the essential charactered of the simulated view.

That said, I believe current LLMs are noticeably simulacra in an important way: they do not appear actively optimize for goals that we attempt to push them towards, instead performing the kind of operations that one would make when trying to achieve that goal.

The canonical example is an LLM presenting a proof, hiding an assumption in a step (which it admits but says is somehow vaguely derivable from the rest/should be assumable) and then declaring QED. We want models that appear to be actively engaged in completing goals.

I am only gesturing at this, so I can't 'prove it' or even make it falsifiable. But the important point here is that, as many have said, nature does not grade on a curve: evolutionarily optimized organisms have to do the thing, or die. LLMs seem to be good at acting towards goals they've seen others act towards, but their actions towards truly novel goals seem performative.

Maybe this perception is just a blip, a little bug in the current tech. Not sure yet.