Bertrand continues to be smart. Do you need a primer in how and why AI works? This is a great overview of the concepts. Like many people, I get excited about the math part. Vector math accelerated by fast computing coupled with my interest in philosophy of language is really flipping cool. It's fun to see how bigger capacities of computing enabled the higher dimensions. However, many of the social implications are elided here, and I'd love to see them addressed.
First, "The Internet" is used as an input, eg Wikipedia. So the sum of human knowledge and culture is used to generate outputs. IMO, if the algorithm is using me as an input, it should pay me for the privilege or be free. IMO, any for profit use of "The Internet" will be inherently exploitative and lead to an end that is not great for human culture. I think this math is really flipping cool but at the very least the algorithms whose training sets use public domain knowledge should be put in the public domain. Or, anyone using public domain as input should pay the value of this (i.e. trillions) into education, poverty, or hunger to make a UBI.
Second, the compute power necessary for these large parameter sets is ginormous. I remember Richard Crandall once complaining to me about how many cycles were being used to raytrace Pixar movies when he barely had enough computing to model smallpox outbreaks so he was interested in Xgrid. That probably wouldn't be so much of an issue now 20 years later. However, I'd like to see a requirement that data centers over [insert correct parameter here] gigaflops be powered by 2x the renewables (i.e. don't just cover your own power consumption but what would have been covered if you didn't exist.) This would encourage development of more efficient renewables and/or more efficient computing for large vector problems. (Made me realize why Nvidia has been so successful at this.)
Third, as we let LLMs reason for us and generate content, we will either save the current state of the models, or train the models on model generated content. We will either get model drift, and no new reasoning in the algorithm, or we will begin to get mad cow disease and our prions will eat ourselves. See - how SEO optimized content ruined the Internet searches returned by google.
At any rate, this is a great half hour if you do not know anything about AI, and if you do, it's a great reducer of all the concepts into byte sizes you can regurgitate for those in your life who don't quite understand math.
Since GPT3 came out, I've been wondering and researching why LLMs work so well. Here are some thoughts: https://lnkd.in/ef3ZGmPU .
Writer of Code at Apple
2moreally makes me miss your presentations at WW :)