AI learning rate and some thoughts

Now that DeepSeek released their new model, I'm thinking again about where AI is heading. For some technologies such as electric cars or batteries, I think I have a reasonable idea of where they are heading thanks to learning curves. Due to price decreases, we will probably see more electric cars, drones, ships, and trucks. Oh, and of course stoves that can boil water in 40 seconds. For AI, I'm not sure. In this blog post, I'll try to estimate the learning curve for AI and see if I can use that to make predictions.

Learning curves, also known as Wright's Law, describe the phenomenon that for some technologies the cost decreases as the number of units produced increases. One of the most famous examples is the cost of computer chips also known as Moore's Law. Although Moore's Law is often associated with the number of transistors on a chip, it is actually about cost as well. From the paper by Gordon E. Moore:

"In 1970, the manufacturing cost per component can be expected to be only a tenth of the present cost."

And Moore was right. In 1961, the price per floating point operation was about $190 billion in inflation-adjusted dollars. In 2023, the price for a floating point operation was about $0.0125. That means that the price went down by about 13 orders of magnitude.

Furthermore, while the price went down, the quality went up. Modern processors are less likely to break down and are more energy efficient.

Another recent example is solar panels. This is what happened to the price of solar panels:

From 2000 to 2023, the price of solar panels has decreased from about 7 $/W to 0.3 $/W. To some, this might sound like a small difference compared to the step from 100 $/W to 10 $/W, but those absolute numbers are not the best way to compare the two. Instead, think about how much panels you can buy for a real-world use-case. For example, a house typically has about 8 kW of solar panels. In 2000 this would cost about 7 $/W * 8 kW = $56,000. In 2023, the same amount of panels would cost only 0.3 $/W * 8 kW = $2,400.

While the price of solar panels is going down exponentially, the amount of solar installations is going up exponentially:

In another blog post, I showed that Tony Seba has correctly predicted that batteries would roughly become 14% cheaper every year:

In this graph, the lines show 12% and 16% cost decline as was predicted by Tony Seba in 2010 and 2014. As you can see, the actual cost decline has indeed been very close to Tony's predictions.

So maybe we can use the same method to estimate the future of AI?

Learning rates

Before going into AI, let's first look at learning rates for other technologies. For solar panels for example, we know that the cost decline occured over \(2023 - 1975 = 48\) years. The rate can then be calculated as follows: \((0.3 / 100)^{1/48} = 0.886\). Just to verify, \(100 * 0.886^{48} ≈ 0.3\).

This means that the costs declines by roughly \(1-0.886=12\%\) every year. So, the learning rate for solar panels is 12%.

Using the same method, I estimated the following numbers with the use of Fermi estimates (also known as order-of-magnitude estimates). The idea of Fermi estimates is that over- and underestimates can cancel each other out when multiplying terms. Next to the cost declines, I also estimated the production growth.

In most cases, the cost went down while the quality went up. For example, modern batteries have much longer lifetimes (and warranties) than older batteries. Also, modern SSDs are much faster than older hard disks.

I'll now speculate a bit on why the production growth is so different for the different technologies. My guess is that the production growth is largely dependent on how much better and cheaper the technology is compared to the alternative. When technologies replace something else, the change goes very rapidly. But at some point, they will reach a point where they have replaced all alternatives and then growth will slow down.

Data storage and computer chips have already replaced all alternatives. In 1953, you could choose between a hard disk drive or magnetic tape. The hard disk drive at some point became cheaper and better than the magnetic tape leading everyone to switch. Now, there is not competition. Most places already use SSDs, so growth is limited to finding new use-cases.

However, batteries are replacing existing use-cases. The benefit batteries have in cars is that they require less maintenance and are about 2-3 times more energy efficient than combustion cars^[10]. It's similar for network transport. It looks like things are still moving over from physical transport to network transport. More and more meetings are held online instead of in person. Even shopping of course is partially replaced by moving pictures of the product over the internet instead of physically going to the store.

AI learning rate

AI is what caused me to write this blog post. I already knew that the price per teraflop of compute power has been going down very rapidly. For example, Bill Dally from Nvidia showed that inference performance per chip has increased by 1000x in the last 10 years. This means a performance increase of about 100% per year while the price per chip has not increased by 1000x.

On top of this, Deepseek released their new model and people have estimated that training cost has decreased from about $500 million to about $5 million. Similarly, their inference pricing is also an order of magnitude lower than OpenAI. It looks like there are two factors that are rapidly reducing the cost of AI. The hardware costs are going down while algorithms are also becoming more efficient.

I was curious how fast the cost of AI is going down. For this, let's use the price of inference:

This means the cost decline per year was 77%^[15]. Also here, the price went down while the quality/performance went up. What the cost decline means is that inference pricing for state-of-the-art models will probably go down to $0.24 per 1M output tokens around Jan 2026, $0.05 per Jan 2027, and $0.01 per Jan 2028. If you keep quality constant, the decrease is about 1000% per year.

As a practical example of what this decrease for state-of-the-art models means, let's say you use an AI copilot in your text editor as a autocomplete or you use AI to generate to starting point for text documents. Then, you would probably generate about 1000 tokens per day^[16]. But say 3000 tokens per day to be on the safe side and to also account for input tokens.

Then the monthly cost is currently around $0.03. Given that it probably saves me about 30 * 30 = 900 minutes per month, this is multiple orders of magnitude cheaper than the thing it replaces, namely me.

Now to question is where this is heading. Based on the reasoning above, I have to re-conclude that AI-powered applications will replace many existing applications. A lot of people currently are trying to use AI, but have a hard time finding use-cases. Based on current stock prices, many people think that these new applications will come out of the big tech. I think that's unlikely.

AI in code editors is my favorite example of this. Microsoft was early with releasing Copilot. Simply put, Copilot is autocomplete for programmers. But while Microsoft had everything it needed to release the best AI code editor, they didn't. Microsoft had the biggest AI infrastructure (Azure), they had the most used code editor (Visual Studio Code), they had the most capable models (via OpenAI), and they were the first to release an AI code editor. Still, Cursor AI came in and many programmers switched to Cursor.

DeepSeek is another example. Microsoft, Google, Meta, OpenAI, and Anthropic all had the most experience and the most resources. Still, DeepSeek came out of the blue with multiple innovations. DeepSeek's CEO, Liang Wenfeng, argues that this is because this is a new technology. According to Wenfeng all you need is small teams with "fresh graduates from top universities, PhD candidates in their fourth or fifth year, and some young people who graduated just a few years ago." My interpretation is that young people are better at finding new solutions to new problems, whereas older and more experienced people are better at improving existing solutions. This is why improvements in the TSMC process need a PhD and years of industry experience, but advances in Gemini at Google were made by Sholto who was only in the field for 1.5 years.

I think that the same will happen with other AI-powered applications. The big companies will probably try to figure out the best applications, but it is likely that new players will come up with new applications that have better AI integrations. As DeepSeek's CEO put it, "Using Internet business logic to discuss future AI profit models is like discussing General Electric and Coca-Cola when Pony Ma [Founder of Tencent; one of the biggest companies in the video game industry] was starting his business." Taleb would probably agree. In a recent interview he said that if you wanted to invest in the internet in 1999, you would probably have invested in Altavista. But then Google came out of nowhere and displaced Altavista. Note also that Google was built by a few young and highly technical people.

As a Google employee predicted in 2023, open source has a good chance of winning. Maybe we will see replacements for Word, Photoshop, Slack, or PowerPoint. Also, and this is more exciting, we will probably see more new/niche applications that are only possible due to AI.

For now, I'll leave it here. I hoped that estimating the learning rate would give me at least some idea of where AI would be heading, but I feel like I still don't know. What I do know is that the learning rate is currently very high. AI is already very cheap and will probably become much cheaper fast.

It's going to be interesting.

Footnotes