I believe for AI we have to step back and look at smaller but important problems that AI can't solve right now. Like basic arithmetic, unlike humans who by grasping concept of arithmetic can extrapolate to any arbitrarily large numbers of operands, neural nets can't do that especially if you're input data is just a bunch of arithmetic pairs and output. But that's changing promising research from Waterloo changes that: https://arxiv.org/abs/2502.16763 . This can be further worked on for other arbitrary algorithms and even languages as they conform to grammatical rules and structure.

Do I think it's similar to how we human minds operate. Probably not but it would be closely approximating it. Bringing determinism in a probabilistic system is challenging but logical thing to do as, our whole classical world is an approximation of quantum mechanics. Consciousness could be a quantum phenomenon like Penrose Hameroff suggested. We are able to move from one system to other system, from Lucas Penrose argument that would mean consciousness could be the only thing allowing this. Brain and neural networks, even language itself is very important for intelligent task. But understanding could very well be an innate property of consciousness. That could explain how human child can do arbitrary arithmetic while neural nets can't neural nets depends on pattern prediction just like human brain, but understanding by virtue of consciousness allows us to transcend the rules.