Achieving Technological Singularity through Analog Computing

At this point, it is almost a cliché to discuss how ChatGPT has transformed our world and strengthened our belief in the power of neural network modeling. We eagerly embraced this technology, hoping for superior results. Nowadays, our focus has shifted to what we can achieve by leveraging neural network models, having witnessed firsthand their capabilities. We are investing increasingly in this technology—our money, our time, our lives.

The essence of neural networks lies in their process: a value is input into the network and, through numerous gates and operations, emerges as something entirely different at the output. To an observer, the intermediate steps may appear random, but each of these steps contributes to the final result.

Randomness often appears as if everything is just random, much like the workings of our brain (if a Homo sapiens is reading this). Our brain also fires what seem like random electrons in its complex structure. It doesn't have to make sense why an electron is where it is at any given moment; what matters is what it will do throughout the process.

This concept is very different from the computers we have designed so far. In a computer, you wouldn't see many electrons seemingly wandering around randomly. Every electron has a predetermined purpose, moving to a predetermined place at a extremely precise predetermined time.

If we think of every computing system as network anything in between input and output is all calculated predetermined and precise. This is contrary to current neural network based models. anything in between input and output is not purposely calculated is not precise in the sense we understand (than what is precise).

Based on this reasoning, I believe that for a true technological singularity to occur, our computing systems must transition from traditional methods to analog computing. In this paradigm, each electron's behavior is not predetermined by us, similar to the concept behind IBM's NorthPole chip (link). I think the entire chain of our computing infrastructure should consist of devices like this. Only then can we achieve an accessible and sustainable technological singularity.

Can technological singularity be reached by current computing approach? Yes but at what cost? who will own it? During the conference I attended a speaker focused on electricity demand of model training. One presenter said that more than half of the cost of largest model that we trained so far was electricity it didn't sounded right and did a quick calculation on the spot.

Cost of an hour of computing in Azure with V100 card is 3.06$ per hour lets say that system consumes 2 kwh. For rest of the infrastructure lets say it consumes 1kwh so total 3 kwh. Lets buy electricity for 30 cents per kwh. 0.3*3 = 0.9$ is the cost of electricity in 3.06$ per hour computing. I was generous with the numbers and even in the most favorable scenario it is not even half of the cost. 

This is very rough calculation just for computation part but proves two points one is that calculations of the presenter favored the electricity sector (which he was affiliated) and I'm gland I have instincts. Second point is even in this calculation cost of electricity in computation is not negligible (not >50%) which proves that current computation approach is not as efficient as it needs to be for technological singularity. I didn't even pointed out that manufacturing techniques used for IBM's NorthPole chip is several years old compared to V100 which utilizes cutting edge technology.

Regardless of all the small details, electricity is a big expense in this current trend and it is caused by our current computing approach. We need to be much more efficient to make it sustainable.

I thought about this during one of the presentations of https://yz2024.gsu.edu.tr/ an AI summit. Had great time. Many thanks to organizers!