Artificial intelligence startup Sakana, based in Tokyo and founded by former Google AI scientists Llion Jones and David Ha, has introduced a groundbreaking AI model architecture known as Continuous Thought Machines (CTM). CTMs are designed to revolutionize AI language models by enhancing flexibility and enabling them to tackle a wider range of cognitive tasks, such as solving complex mazes and navigation challenges without positional cues or spatial embeddings.
Unlike traditional Transformer models that rely on fixed, parallel layers to process inputs simultaneously, CTMs unfold computation over steps within each artificial neuron. Each neuron retains a short history of its previous activity, utilizing this memory to determine when to activate again. This unique internal state allows CTMs to dynamically adjust the depth and duration of their reasoning based on the complexity of the task at hand, making each neuron more informationally dense and complex compared to Transformer models.
Sakana has published a paper on arXiv, launched a microsite, and made their Github repository available for further exploration of CTMs. The model’s ability to reason progressively, adaptively allocate compute resources, and offer interpretability makes it a promising development in the AI landscape. While CTMs are not yet optimized for commercial deployment, their open-source nature and availability of training scripts and tools make them accessible for researchers and engineers looking to experiment with this innovative architecture.
