为什么人工智能无法自主发现？——与Yann LeCun对话

我们都在讨论这里可能存在收益递减的问题。

And we both are talking about how, like, there's, you know, potentially diminishing returns here.

然后如果出现你提到的那种时间线不匹配的情况会怎样。

And then what happens if there's that timeline mismatch, like you mentioned.

是的。

Yeah.

这是我关于这个问题的最后一个问题了，因为我们还有很多其他内容要讨论，但我觉得时间线不匹配可能对你个人有特殊意义。

And this is the last question I'll ask about it because I feel like we have so much else to cover but I feel like timeline mismatches that might be personal to you.

你和我九年前第一次交谈，现在想来真是疯狂，九年前你对AI应该如何构建就有想法，却连参加会议的席位都争取不到。后来随着算力的提升，这些想法开始奏效，整个AI领域便基于你与Bengeo和Hinton共同研究的理念腾飞而起。

You and I first spoke nine years ago which is crazy now, nine years ago and you know about how in the early days you had an idea for how AI should be structured and you couldn't even get a seat at the conferences and then eventually with the when the right amount of compute came around those ideas started working and then the entire AI field took off based off of your idea that you worked on with Bengeo and Hinton.

还有一群其他人。

And a bunch of others.

以及许多其他人，但为了效率起见，我们会说去查查资料吧。

And many others and but for the sake of efficiency, we will say go look it up.

就是在讨论那些时间线错配的情况，当AI领域出现过度炒作时刻，比如你刚提到的专家系统，它们并未如人们预期那样发展，AI领域就会进入所谓的'AI寒冬'。

Just talking about those mismatched timelines when there have been over hyped moments, in the AI field, maybe with expert systems that you were just talking about and they don't pan out the way that people expect, the AI field goes into what's called AI winter.

嗯，会有反弹的。

Well there's a backlash.

是的。

Yep.

没错。

Correct.

那么，如果我们正面临这种时间线错配的时刻，考虑到当前的投资规模、主要训练方法可能出现的收益递减，再加上股市似乎正在经历低迷，你是否担心现在可能会出现又一次的寒冬期？

And so if we're gonna if we are potentially approaching this moment of mismatched timelines do you fear that there could be another winter now given the amount of investment given the fact that there's going to be potentially diminishing returns with the main way of training these things and maybe we'll add in the fact that the market is is the stock market looks like it's going through a bit of a downturn right now.

这可能是我们讨论中第三重要的变量，0但必须纳入考量。

Now that's a variable probably the third most important variable of what we're talking about, but it has to factor.

所以，是的，我认为...我是说，时机问题确实存在，但正如我之前所说，如果我们认为仅0, 仅通过,,),, 但如果我们试图更深入一点，就像我之前说的，如果你认为我们仅仅通过训练更多数据和扩大LLMs就能达到人类水平的AI，那你就错了。

So, yeah, I I think I mean, there's certainly a question of timing there, but I think if we try to dig a little bit deeper, as I said before, if you think that we're going get to human level AI by just training on more data and scaling up LLMs, you're making a mistake.

所以如果你是投资者，投资了一家告诉你'我们仅通过训练更多数据和一些小技巧就能达到人类水平和博士水平AI的公司。

So if you're an investor and you invested in a company that told you we're going to get to human level AI and PhD level by just training on more data and with a few tricks.

我不知道你是否会用你的衬衫，, 但你可能不知道，，但可能不是一个好主意。

I don't know if you're going to use your shirt, but that was probably not a good idea.

然而，关于如何推进并构建能够实现所有智能动物和人类能力（而当前我们的系统尚不具备这些能力）的系统，已有一些构想。

However, there are ideas about how to go forward and have systems that are capable of doing what every intelligent animal and human are capable of doing and that currently our systems are not capable of doing.

我指的是理解物理世界、拥有持久记忆以及具备推理和规划的能力。

And I'm talking about understanding the physical world, having persistent memory, and being able to reason and plan.

这四个特性是必须具备的。

Those are the four characteristics that need to be there.

这就要求系统能够获取常识，能够从视频等自然传感器中学习，而不仅仅是依赖人类产生的文本数据。

And that require systems that can acquire common sense, that can learn from natural sensors like video as opposed to just text, just human produced data.

这是一个巨大的挑战。

And that's a big challenge.

事实上，我已经就此谈论多年，指出这正是挑战所在。

I mean, I've been talking about this for many years now and saying this is where the challenge is.

这是我们必须解决的问题。

This is what we have to figure out.

我和我的团队，以及一些听从我建议的人，正在这条道路上取得，例如开发 systems that can be trained to understand

And and my group and I have or people working with me and others who have listened to me are making progress along along this line of systems that can be trained to understand how the world works on video, for example, systems that can use mental models of how the physical world works to plan sequences of actions to arrive at a particular goal.

所以我们已经有这类系统的初步成果。

So we have kind of early results of these kind of systems.

DeepMind和其他一些大学的研究人员也在从事类似的工作。

And there are people at DeepMind working on similar things, and there are people in various universities working on this.

问题在于，何时这些研究才能从展示新架构能力的论文，发展为可大规模应用、能自主解决新问题的实用架构。

So the question is, when is this going to go from interesting research papers demonstrating a new capability with a new architecture to architectures at scale that are practical for a lot of applications and can find solutions to new problems without being trained to do it, etcetera.

这不会在未来三年内实现，但可能在三年到五年之间达成。

And it's not going to happen within the next three years, but it may happen between three to five years, something like that.

这与我们看到的投资增长趋势大致吻合。

And that kind of corresponds to the sort of ramp up that we see in investment.

这是第一点。

Now whether other so that's the first thing.

第二点重要的是，不会存在某个公司或团队能发明一个神奇的万能解决方案。

Now the second thing that's important is that there's not going to be one secret magic bullet that one company or one group of people is going to invent that is going to just solve the problem.

这将需要多种思路、大量努力，以及某些可能不被所有人认同的基本原则，其中一些方向最终可能会走入死胡同。

It's going to be a lot of different ideas, a lot of effort, some principles around which to base this that some people may not subscribe to and will go in a direction that will turn out to be a dead end.

所以不会出现一个分水岭，在那之前没有通用人工智能，之后突然就有了。

So there's not going to be, like, a day before which there is no AGI and after which we we have AGI.

这不会是一个标志性事件。

This is not going to be an event.

这将是一个持续的概念演进过程，随着时间的推移，规模会越来越大，性能会越来越好。

It's going to be continuous conceptual ideas that, as time goes by, are going to be made bigger and to scale and gonna work better.

而且它不会来自某个单一实体。

And it's not gonna come from a single entity.

它将来自全球范围内的整个研究社区。

It's gonna come from the entire research community across the world.

那些分享研究成果的团队会比封闭研究的团队发展得更快。

And the people who share their research are gonna move faster than the ones that don't.

所以如果你认为某个五人初创公司掌握了通用人工智能的秘密，并打算投资50亿美元给他们，那你就犯了大错。

And so if you think that there is some start up somewhere with five people who has discovered the secret of AGI and you should invest 5,000,000,000 in them, you're making a huge mistake.

你知道，简，首先我一直很享受我们的对话，因为总能得到一些真知灼见。我记得上次谈话后，我经常回想那次对话，想着'简是这样说的'。

You know, Jan, first of all, I always enjoy our conversations because we start to get some real answers, and I remember even from our last conversation, was just always looking back to that conversation saying, This is what Jan says.

这就是其他人都在说的。

This is what everybody else is saying.

我很确定这是基础点，这一直是正确的，我知道我们这次也会这样做。

I'm pretty sure that this is the grounding point, and that's been correct, and I know we're gonna do that with this one as well.

现在你为我铺垫了两个有趣的话题线索，我们将在接下来的对话中展开讨论。

And now you've set me up for two interesting threads that we're gonna pull out as we go on with our conversation.

第一是对物理学和现实世界的理解，第二是开源。

First is the understanding of physics and the real world, and the second is open source.

所以我们稍后回来再继续这个话题。

So we'll do that when we come back right after this.

你想吃得更好，但既没时间也没精力去实现。

You wanna eat better, but you have zero time and zero energy to make it happen.

Factor不需要你备餐或照着食谱做。

Factor doesn't ask you to meal prep or follow recipes.

它直接解决了整个问题。

It just removes the entire problem.

两分钟，真材实料，轻松搞定。

Two minutes, real food, done.

还记得你想做健康餐却点了披萨那次吗？

Remember that time where you wanted to cook healthy but ordered pizza?

你并不是健康饮食的失败者。

You're not failing at healthy eating.

你只是没法每天多挤出三小时而已。

You're failing at having three extra hours every night.

Factor的餐点由厨师现做、营养师设计，并直送到你家门口。

Factor is already made by chefs, designed by dietitians, and delivered to your door.

你只需加热两分钟就能享用。

You heat it for two minutes and eat.

内含优质蛋白、多彩蔬菜、全天然食材和健康脂肪——这些本该是你有时间就会自己做的。

Inside, there are lean proteins, colorful vegetables, whole food ingredients, healthy fats, the stuff you'd make if you had the time.

立即访问factormeals.com/bigtech50off，并使用优惠码bigtech50off，首单立享5折外加全年免费早餐。

Head to factormeals.com/bigtech 50 off and use code big tech 50 off to get 50% off your first factor box plus free breakfast for one year.

该优惠仅适用于使用该代码并购买符合条件的自动续订订阅的新Factor客户。

The offer is only valid for new factor customers with the code and qualifying auto renewing subscription purchase.

通过Factor让健康饮食变得更简单。

Make healthier eating easy with Factor.

现在我们再次欢迎Jan Lecun。

And we're back here with Jan Lecun.

他是Meta的首席AI科学家，也是图灵奖得主，我们非常荣幸能第三次邀请到他在我们的节目中。

He is the chief AI scientist at Meta and the Turing Award winner that we're thrilled to have on our show luckily for the third time.

我想和你聊聊物理学，Jan，因为在科技播客历史上有个著名时刻——当然这个‘著名’是对于我们听众而言的。

I wanna talk to you about physics, Jan, because there's sort of this famous moment in big technology podcast history, and I say famous with our listeners.

我不确定这个梗是否出圈了，但你当时让我问ChatGPT：如果我双手水平拿着一张纸，然后松开左手，会发生什么？

I don't know if it really extended beyond, but you had me write to chat GPT if I hold a paper horizontally with both hands and let it let the go let go of the paper with my left hand, what will happen?

我输入问题后，它煞有介事地写出一堆物理原理，说纸张会向左飘。我念答案时深信不疑，你却指出AI在胡编而我居然信了。

And I write it and it convincingly says like it writes the the physics will will happen and the paper will float towards your left hand and I read it out loud convinced and you're like that thing just hallucinated and you believed it.

事实确实如此。

That is what happened.

所以听着，已经过去两年了。

So listen it's been two years.

我今天又测试了ChatGPT。

I put the test to chatty PT today.

它说当你用左手松开纸张时，重力会使纸张左侧下坠，而仍被右手握住的右侧保持原位，这会产生一个支点效应，纸张会围绕右手握持点旋转。

It says when you let go of the paper with your left hand gravity will cause the left side of the paper to drop while the right side still held up but your right hand remains in place this creates a pivot effect where the paper rotates around the point where your right hand is halting it.

所以现在它答对了。

So now it gets it right.

它吸取了教训。

It learned the lesson.

你知道吗？

You know?

很有可能这是OpenAI雇人专门解决的问题，他们输入了这个问题和答案，系统通过微调掌握了正确答案。

It's quite possible that this you know, some someone hired by OpenAI to solve the problem was fed that question and sort of fed the answer, and the system was fine tuned with the answer.

我是说，显然你能想象这类问题有无限多种。

I mean, obviously, you can imagine an infinite number of such questions.

这就是所谓的NLM后训练变得昂贵之处——你需要覆盖多少这类问题，才能使系统基本涵盖人们可能提出的90%或95%的问题。

And this is where the so called post training of NLM becomes expensive, which is that how much coverage of all those type of questions do you have to do for the system to basically cover 90% or 95% or whatever percentage of all the questions that people may ask it.

但你知道，这里存在长尾效应，你无法训练系统回答所有可能的问题，因为本质上问题是无限的。

But there you know, it's there's a long tail, and there is no way you can train the system to answer all possible questions because there is an essentially infinite number of them.

而且系统无法回答的问题远比它能回答的问题多得多。

And and there is way more questions the system cannot answer that than questions it can it can answer.

你无法覆盖所有可能的训练问题集合，明白吗，训练集中的训练集。

You cannot cover the set of all possible training, you know, questions in the training set the training set.

对。

Right.

因为我想起我们上次谈话时你说过，像纸张这种物理行为如果没有被广泛记录在文本中，模型就不知道如何处理——除非文本中有相关描述，否则模型不会具备那种对现实世界的内在理解。

So because I think our conversation last time was saying, you you said that because these actions of, like, what's happening with the paper, if you let go of it with your hand has not been covered widely in text, the model won't really know how to handle it because unless it's been covered in text, the model won't have that understanding, won't have that inherent understanding of the real world.

我一度接受这个观点，但后来想，不如试试生成一些AI视频吧。

And I've kind of gone with that for a while, then I said, you know what, let's let's try to generate some AI videos.

我在AI视频中发现一个有趣现象：它们对物理世界运作方式存在某种理解。这让我想起九年前我们第一次见面时你说的——最难解决的问题之一就是问AI‘如果你把笔垂直放在桌上然后松手会发生什么？’

And one of the interesting things that I've seen with the AI videos is there is some understanding of how the physical world works there in a way that in our first meeting nine years ago you said one of the hardest things to do is you ask an AI what happens if you hold a pen vertically on a table and let go?

它会掉下来吗？

Will it fall?

这其中存在着难以置信的大量可能性变化，AI要理解这些极其困难，因为它本质上并不理解物理规律。

And there's like an unbelievable amount of permutations that can occur and it's very very difficult for the AI to figure that out because it just doesn't inherently understand physics.

但现在你看看像Sora这样的系统，你只要说‘给我看一个人坐在椅子上踢腿的视频’，就能得到这样的视频。

But now you go to something like Sora and you say show me a a video of a man sitting on a chair kicking his legs, and you can get that video.

视频里的人会坐在椅子上踢腿，腿部关节会正常弯曲，而不会出现腿从关节处脱落之类的情况。

And the person sits on the chair, and they kick their legs, and the legs, you know, don't fall out of their sockets or stuff they bend at the joints.

而且它们没有三条腿。

And they don't have three legs.

而且它们没有三条腿。.

And they don't have three legs.

那么这是否表明这些大型语言模型的能力有所提升？?

So wouldn't that suggest an improvement of the capabilities here with these large large models?

不。

No.

为什么？

Why?

因为那些视频生成系统制作的视频里，你倒一杯酒时，酒会悬在空中、飞走或直接消失等等。

Because you still have those videos produced by those video generation system where, you know, you spill a glass of wine and and the wine, like, floats in the air or, like, flies off or disappears or whatever.

当然，针对每个具体场景，你总能收集更多相关数据来训练模型处理它。

So of course, for every specific situation, you can always collect more data for that situation and then train your model to handle it.

但这并非真正理解底层现实。

But that's not really understanding the underlying reality.

这只是用越来越庞大的数据量来弥补理解力的不足。

This is just compensating the lack of understanding by increasingly large amounts of data.

孩子们仅需极少数据就能理解重力这类简单概念。

Children understand simple concepts like gravity with a surprisingly small amount of data.

实际上可以做个有趣的计算，我之前也提到过。

So in fact, there is an interesting calculation you can do, which I've talked about previously before.

以典型的大语言模型为例，训练数据达3万亿token量级，约3×10¹³个token。

But if you take LLN, typical LLN trained on 30,000,000,000,000 tokens, something like that, right, three ten to the 13 tokens.

每个标记大约三个字节，所以大约是0.9乘以10的14次方个标记。

The token is about three bytes, so that's point nine ten to the 14 tokens.

我们四舍五入一下，就说是10的14次方个标记吧。

Let's say 10 to the 14 tokens to to round this up.

这些文本内容，我们任何人可能都需要大约四十万年的时间才能读完。

That text would take any of us probably some on the order of four hundred thousand years to read.

对吧？

Right?

没问题。

No problem.

每天读十二个小时的话。

At twelve hours a day.

明白吗？

Okay?

而一个四岁孩子醒着的时间总共才16,000小时。

Now a four year old has been awake a total of 16,000.

你可以乘以3600得到秒数，然后就能计算出通过视神经传递到视觉皮层的数据量。

You can multiply by 3,600 to give number of seconds, and then you can put a number on how much data has gotten to your visual cortex through the optic nerve.

每条视神经（我们共有两条）每秒大约传输1兆字节的数据。

Each optic nerve, we have two of them, carries about one megabyte per second roughly.

所以是每秒2兆字节乘以3600再乘以16000。

So it's two megabytes per second times 3,600 times 16,000.

这大约就是10的14次方字节。

And that's just about ten fourteen bytes.

因此四年间，一个孩子通过视觉或触觉接收的数据量，已经相当于最大的语言模型系统。

So in four years, a child has seen through vision or touch, for that matter as much data as the biggest LMS.

这清楚地表明，仅靠文本训练是无法达到人类水平的人工智能的。

And and it tells you clearly that we're not gonna get to human level AI by just training on text.

文本作为信息来源的丰富程度远远不够。

It's just not a rich enough source of information.

顺便说，16000小时的视频量其实并不算多。

And by the way, sixteen thousand hours is not that much video.

相当于30分钟的YouTube上传内容。

It's thirty minutes of YouTube uploads.

明白吗？

Okay?

嗯哼。

Mhmm.

我们可以相当轻松地获取这些数据。

We can get that pretty easily.

在九个月的时间里，婴儿已经接收了大约10的13次方字节的数据，这其实也不算多。

Now in nine months, Baby has seen, you know, let's say, 10 to the 13 bytes or something, which is not not much, again.

在这段时间里，婴儿基本上掌握了我们所知的所有直觉物理知识——动量守恒、重力、物体不会凭空消失、即使被藏起来也依然存在等概念。

And in that time, Baby has learned basically all of intuitive physics that we know about, conservation of momentum, gravity, conservation of momentum, the fact that objects don't spontaneously disappear, the fact that they still exist even if you hide them.

我的意思是，我们在生命最初几个月里就学会了关于世界的各种基础认知。

I mean, there's all kinds of stuff, very basic stuff that we learn about the world in the first few months of life.

这正是我们需要用机器复现的学习方式——理解世界中哪些是可能和不可能的，预判行为后果，从而规划行动序列以实现特定目标。

And this is what we need to reproduce with machine, this type of learning of figuring out what is possible and impossible in the world, what will result from an action you take so that you can plan a sequence of actions to arrive at a particular goal.

这就是世界模型的概念。

That's the idea of world model.

现在联系到你关于视频生成系统的问题，训练越来越好的视频生成系统是解决这个问题的正确方法吗？

And now connected with your question about video generation systems, is the right way to approach this problem to train better and better video generation systems?

我对这个问题的回答是绝对不行。

And my answer to this is absolutely no.

理解世界的问题并不需要通过像素级视频生成的解决方案。

The problem of understanding the world does not go through the solution to generating video at the pixel level.

明白吗？

Okay?

我不需要知道如果我拿起这杯水并打翻它，我无法完全预测水的具体流动路径、在桌上的形状变化以及会发出什么声音等等细节。

I don't need to know if I take this glass of this cup of water and I spill it, I cannot entirely predict you know, the exact path of that the water will will follow on the table and what shape it's gonna take and all that stuff, what noise it's going to make.

但在某个抽象层面上，我可以预测水会洒出来。

But at a certain level of abstraction, I can make a prediction that the water will spill.

明白吗？

Okay?

而且可能会弄湿我的手机和其他东西。

And it'll probably make my phone wet and everything.

所以我无法预测所有细节，但我可以在某种抽象层面上进行预测。

So I can't predict all the details, but I can predict at some level of abstraction.

我认为这是一个非常关键的概念：如果一个系统要学习理解世界并掌握其运作方式，它需要能够学习世界的抽象表征，从而做出这些预测。

And I think that's really a critical concept, the fact that if you want a system to be able to learn to comprehend the world and understand how the world works, it needs to be able to learn an abstract representation of the world that allows you to make those prediction.

这意味着这些架构将不具备生成能力。

And what that means is that those architectures will not be generative.

对。

Right.

稍后我想讨论你的解决方案，但首先——我们之间的对话怎么能没有演示呢？

And I wanna get to your solution here in a moment, but I just wanted to also like, what would a conversation between us be without a demo?

我想展示一下，做视频时我会把这个放上屏幕——这是段让我相当自豪的视频。

So I wanna just show you I'm gonna put this on the screen when we do the video but there's this is a video I was pretty proud of.

视频里有个家伙坐在椅子上踢腿，结果腿还连在身上，我当时就觉得：哇，这技术真的在进步！然后我又尝试让一辆车冲进干草堆.

I got this guy sitting on a chair kicking his legs out and the legs stay attached to his body and I was like alright this stuff is making real progress and then I said can I get a car going into a haystack and so it's two bales of haystacks and then a haystack magically emerges from the hood of a car that's stationary And I just said to myself, okay?

简又赢了。.

Jan wins again.

这辆车确实不错.

It's it's a nice car, though.

是的.

Yeah.

我的意思是，关键在于这些系统已经通过大量人类数据进行了微调，因为你知道，这就是人们所}

I mean, the thing is those systems have been fine tuned with a huge amount of data for humans because, you know, that's that's what people are asking.

大多数视频，它们要求执行。

Most videos, they ask to do.

因此有大量人类进行各种活动的数据来训练这些系统。

So there is a lot of data of humans doing various things to train those systems.

这就是为什么它对人类有效，但对于训练该系统的人未曾预料到的情况则无效。

So that's why it works for humans, but not for a situation that the the people training that system had not anticipated.

所以你说模型要理解现实世界就不能是生成式的。

So you said that the model can't be generative to be able to understand the real world.

没错。

That's right.

你正在研究一个叫vJeppa的东西。

You are working on something called vJeppa.

是Jepa。

Jepa.

Jepa。

Jepa.

对吧？

Right?

V代表视频。

V is the video.

你们还有用于图像的ijeppa。

You also have ijeppa for images.

对。

Right.

我们有适用于各种场景的Jepa。

That is We have Jepa for all kinds of stuff.

文本也是。

Text also.

还有文本。

And text.

那么请解释这将如何解决让机器能够抽象表示现实世界中发生的事情的问题。

So explain how that will solve the problem of being able to allow a machine to abstractly represent what is going on in the real world.

好的。

Okay.

近年来AI的成功，尤其是自然语言理解和聊天机器人，在某种程度上也包括计算机视觉，其关键在于自监督学习。

So what has made the success of AI and particularly natural language understanding and chatbot in the last few years, but also to some extent, computer vision is self supervised learning.

那么什么是自监督学习？

So what is self supervised learning?

就是获取一个输入——无论是图像、视频、文本还是其他——以某种方式破坏它，然后训练一个大型神经网络来重建它，基本上是恢复其未损坏的版本、未失真的版本，或者通过采取行动可能产生的转换版本。

It's take an input, be it an image, a video, a piece of text, whatever, corrupt it in some way, and train a big neural net to reconstruct it, basically recover the uncorrupted version of it or the undistorted version of it or a transformed version of it that would result from taking an action.

明白吗？

Okay?

也就是说，比如在文本处理中，取一段文字，删掉部分单词，然后训练一个大型神经网络来预测缺失的单词。

And, you know, that that would mean, for example, in the context of text, take a piece of text, remove some of the words, and then train some big neural net to predict the words that are missing.

对于图像处理，则去掉图像的某些部分，然后训练大型神经网络来恢复完整图像。

Take an image, remove some pieces of it, and then train a big neural net to recover the full image.

视频处理则是去掉片段，训练神经网络预测缺失内容。

Take a video, remove a piece of it, train a neural net to predict what's missing.

明白吗？

Okay?

所以大语言模型是这种方法的特例，你拿一段文本，训练系统去重现这段文本。

So LLMs are a special case of this where you you you take a a text, and you train the system to just reproduce the text.

你不需要破坏文本，因为系统的设计方式让它只能根据左侧的标记来预测文本中的某个特定单词或标记。

And you don't need to corrupt the text because the system is designed in such a way that to predict one particular word or token in the text, it can only look at the tokens that are to the left of it.

懂了吗？

Okay?

因此，实际上，系统的架构中硬编码了它无法通过现在和未来来预测现在这一事实。

So so in effect, the system has hardwired into its architecture the fact that it cannot look at the present and the future to predict the present.

它只能回顾过去。

It can only look at the past.

明白吗？

Okay?

但基本上，就是训练这个系统在其输出中重现其输入。

But basically, train that system to just reproduce its input on its output.

懂了吗？

Okay?

所以这种架构被称为因果架构。

So this kind of architecture is called a causal architecture.

这就是大型语言模型（LLM）的本质。

And this is what an LLM is, a large language model.

全世界所有的聊天机器人都是基于这个原理构建的。

That's what all the chatbots in the world are are based on.

取一段文本并训练系统，使其在输出中重现这段文本。

Take a piece of text and train the system to just reproduce that piece of text on its output.

而要预测某个特定单词时，它只能查看该单词左侧的内容。

And to predict a particular word, it can only look at the word to the to the left of it.

因此现在你得到的系统是：给定一段文本，能够预测该文本后续的单词。

And so now what you have is a system that, given a piece of text, can predict the word that that follows that text.

你可以将这个预测出的单词移入输入，然后预测第二个单词。

And you can take that that word that is predicted, shift it into the input, and then predict the second word.

嗯。

Mhmm.

将其移入输入，预测第三个单词。

Shift that into the input, predict the third word.

这被称为自回归预测。

That's called autoregressive prediction.

这并不是一个新概念。

It's not a new concept.

非常古老。

Very old.

所以，你看，自监督学习并不是为了完成某项特定任务而训练系统，而是为了捕捉数据的内在结构。

So, you know, self supervised learning does not train to do a particular does not train a system to accomplish a particular task other than capture the internal structure of the of the data.

它完全不需要人工标注。

It doesn't require any labeling by by a human.

明白吗？

Okay?

那么拿这两张图片来说。

So apply these two images.

取一张图片，遮掉其中一部分，比如若干个图像块，然后训练一个大型神经网络来重建缺失部分。

Take an image, mask a chunk of it, like a bunch of patches from it, if you want, and then train a big neural net to reconstruct what is missing.

现在就可以用系统学习到的图像内部表征作为后续下游任务的输入，比如图像识别、分割等等，虽然效果有限。

And now use the internal representation of the image learned by the system as input to a subsequent downstream task for, I don't know, image recognition, segmentation, whatever it is, it works to some extent, but not great.

所以我们在FAIR项目里就有一个这样的大型计划。

So there's a big project like this to do this at fair.

它被称为MAE（掩码自编码器）。

It's called MAE, masked autoencoder.

这是去噪自编码器的一种特例，而后者正是我提出自监督学习理念的通用框架基础。

It's a special case of denoising autoencoder, which itself is the sort of general framework from which I derive this idea of self supervised learning.

所以效果不是很好。

So it doesn't work so well.

还有各种方法，比如应用到视频领域——我研究这个已经快二十年了——可以截取视频片段，训练系统预测视频后续内容。

And there's various ways to you know, if you apply this to video, also, I've been working on this for almost twenty years now, take a video, show just a piece of the video, and then train the system to predict what's going happen next in the video.

所以这和文本处理是同样的思路，只是应用于视频。

So same idea as for text, but just for for video.

而且这种方法效果也不太好。

And that doesn't work very well either.

那么为什么它对文本有效而对视频无效呢？

And the reason it doesn't work why does it work for text and not for video, for example?

答案是预测文本中下一个单词要容易得多。

And the answer is it's easy to predict a word that comes after a text.

你无法准确预测特定文本后接哪个词，但可以生成类似词典中所有可能单词或所有可能标记的概率分布。

You cannot exactly predict which word follows a particular text, but you can produce something like a probability distribution over all the possible words in a dictionary, all the possible tokens.

可能的标记数量大约只有10万个左右。

There's only about 100,000 possible tokens.

所以你只需生成一个包含10万个不同数字的大向量，这些数字均为正数且总和为一。

So you just produce a big vector with 100,000 different numbers that are positive and sum to one.

好的。

Okay.

那么，对于视频中所有可能的帧或图像中所有缺失部分，你打算如何表示其概率分布呢？

Now what are you going to do to represent a probability distribution over all possible frames in a video or all possible missing parts of an image?

我们尚不知如何妥善处理这个问题。

We don't know how to do this properly.

事实上，在高维连续空间中表示分布在数学上是难以处理的。

In fact, it's mathematically intractable to represent distributions in high dimensional continuous spaces.

明白了吗？

Okay?

我们不知道如何以一种实用的方式来做这件事，如果你想的话。

We don't know how to do this in a kind of useful way, if you want.

而且我尝试了很长时间，你懂的，为视频做这件事。

And so and I've tried to, you know, do this for video for a long time.

因此，这就是为什么那些使用生成模型进行自监督学习的想法至今未能成功的原因。

And so that is the reason why those ideas of self supervised learning using generative models have failed so far.

这也是为什么，你知道，试图训练一个视频生成系统作为理解世界运行方式的方法。

And this is why, you know, using a you know, trying to train a video generation system as a way to understand to get a system to understand how the world works.

这就是为什么它无法成功。

That's why it's it can't succeed.

那么替代方案是什么？

So what's the alternative?

替代方案是一种非生成式架构，我们称之为JPA，即联合嵌入预测架构。

The alternative is something that is not a generative architecture, which we call JPA, so that means Joint Embedding Predictive Architecture.

我们知道这比尝试重建效果要好得多。

And we know this works much better than attempting to reconstruct.

因此，我们多年来在图像表征学习方面取得的实验成果表明：与其对图像进行损坏并尝试重建图像，我们采用原始完整图像及其损坏版本。

So we've had experimental results on learning good representations of images going back many years where instead of taking an image, corrupting it, and attempting to reconstruct this image, we take the original full image and the corrupted version.

我们将两者都通过神经网络处理。

We run them both through neural nets.

这些神经网络会生成这两张图像（原始图像和损坏图像）的表征。

Those neural nets produce representations of those two images, the initial one and the corrupted one.

然后我们训练另一个神经网络（预测器），从损坏图像的表征预测完整图像的表征。

And we train another neural net, a predictor, to predict the representation of the full image from the representation of the corrupted one.

嗯。

Mhmm.

好的。

Okay.

如果你成功训练了一个这种类型的系统，它并不是为了重建任何东西而训练的。

And if you train a system if you're successfully trained a system of this type, this this is not trained to reconstruct anything.

它只是被训练来学习一种表征，以便你可以在表征层内进行预测。

Just trained to learn a representation so that you can make prediction within the representation layer.

你必须确保表征层尽可能多地包含输入信息，这实际上正是难点所在。

And you have to make sure that the representation contains as much information as possible about the input, which is why it's difficult, actually.

这就是训练这类系统的困难之处。

That's the difficult part of training those systems.

因此这被称为JEPA（联合嵌入预测架构）。

So that's called a JEPA, Joint Embedding Predictive Architecture.

要训练系统学习图像的良好表征，这些联合嵌入架构比通过重建训练的生成式架构效果更好。

And to train a system to learn good representations of images, those joint embedding architectures work much better than the ones that are generative that are trained by reconstruction.

现在我们也有了适用于视频的版本。

And now we have a version that works for video, too.

我们选取一段视频。

So we take a video.

通过遮蔽大片段来破坏它。

We corrupt it by masking a big chunk of it.

将完整视频和被破坏的视频输入完全相同的编码器进行处理。

We run the full video and the corrupted one through encoders that are identical.

然后同时，我们训练一个预测器，从部分视频预测完整视频的表征。

And then and simultaneously, we train a predictor to predict the representation of the full video from the partial one.

当系统学习到的视频表征被输入到一个训练用于识别视频中发生的动作、或判断视频是否可能等任务的系统时，其表现实际上相当出色。

And the representation that the system learns of videos, when you feed it to a system that you train to tell you, for example, what action is taking place in the video or whether the video is possible or impossible or things like that, it actually works quite well.

这很酷。

That's cool.

所以它提供了那种抽象思维

So it gives that abstract thinking

是的。

Yeah.

在某种程度上。

In a way.

对。

Right.

而且我们有实验结果表明，这种联合嵌入训练——我们有好几种方法可以实现这一点。

And and we have experimental result that shows that this joint embedding training, we have several methods for doing this.

其中一种叫DINO，另一种叫VCREG，还有一种也叫VCREG，还有一种是IGPA，这是一种蒸馏方法。

There's one that's called DINO, another one that's called VCREG, another one that's called VCREG, another one that's called IGPA, which is sort of a distillation method.

所以我们有几种不同的方法来解决这个问题，但其中一种将引导出一个基本方案，为我们提供训练这些JPAW架构的通用方法。

And so we had several different ways to approach this, but one of those is going to lead to a recipe that basically gives us a general way of training those JPAW architectures.

好的。

Okay.

所以它不是生成的，因为系统并未被训练来重建输入的部分内容。

So it's not generated because the system is not trained to regenerate the part of the input.

它被训练用于生成输入的一种抽象表示。

It's trained to generate a representation, an abstract representation of the input.

这样你就能忽略输入中所有真正不可预测的细节。

And what that allows you to do is to ignore all the details about the input that are really not predictable.

比如，你把笔竖着放在桌上，松手时无法预测它会倒向哪个方向。

Like, you know, the pen that you put on the table vertically, and when you let it go, you cannot predict in which direction it's gonna fall.

但在某些抽象层面上，你可以说笔会倒下——它正在倒下。

But at some abstract level, you can say that the pen is gonna fall It's falling.

对，但不表示具体方向。

Right, without representing the direction.

这就是JetPa的核心思想。

So so that's the that's the idea of JetPa.

我们开始在一些系统上取得不错的成果。

And and we're starting to have, you know, good results on sort of having systems.

比如VJEPPA系统，就是在大量自然视频上训练的。

So the VJEPPA system, for example, is trained on natural lots of natural videos.

然后你可以给它展示不可能的视频，比如物体消失或变形的视频。

And then you can show it a video that's impossible, like a video where, for example, an object disappears or changes shape.

明白吗？

Okay?

你可以用游戏引擎之类的工具生成这种视频。

You can generate this with a game engine or something.

或者像球体滚动，滚到屏幕后停止这样的场景。

Or a situation where you have a ball rolling, and it rolls and it stops behind a screen.

然后屏幕降下来，球却不见了。

And then the screen comes down, and the ball is not there anymore.

对。

Right.

明白吗？

Okay?

就是类似这样的情况。

So things like this.

然后你测量系统的预测误差。

And you measure the prediction error of the system.

所以这个系统是被训练来预测的。

So the the system is trained to predict.

对吧？

Right?

不一定在时间上，而是基本上预测视频的连贯性。

And not necessarily in time, but but, like, basically, to predict, you know, the the the sort of coherence of the video.

因此你在给系统播放视频时测量预测误差

And so you you measure the prediction error as you show the the video to the system.

当出现不可能的情况时，预测误差就会飙升

And when something impossible occurs, the prediction error goes goes through the roof.

这样你就能检测系统是否整合了某些概念，比如知道什么是物理上可能的、什么是不可能的，而仅仅通过物理可能的自然视频训练

And so you can detect if the system has integrated some idea of what, you know, is possible physically or what's not possible, but just being trained with physically possible natural videos.

这确实非常有趣

So that that's really interesting.

这算是系统获得某种程度常识的第一个迹象

That's sort of the first hint that a system has acquired some level of common sense.

是的

And Yes.

我们也有所谓动作条件的这类系统版本

We have versions of those systems also that are so called action conditions.

基本上我们有一些视频片段或图像，展示时间t的世界状态，然后执行某个动作，比如机械臂移动之类的

So, basically, we have things where we have a chunk of video or an image of, you know, the state of the world at time t, and then an action is being taken like, you know, robot arm is being moved or whatever.

当然，我们随后可以观察这一行动产生的结果。

And then, of course, we can observe the the result resulting from this action.

所以当我们用这种方式训练JEPPA时，模型基本上可以说：这是时间t时的世界状态。

So now what we have when when we train a JEPPA with this, the the model basically can say, here is the state of the world at time t.

这是你可能采取的一个行动。

Here is an action you might take.

我可以预测在这个抽象空间表示下，时间t加一时世界的状态。

I can predict the state of the world at time t plus one in this abstract representation of space.

这是对世界运行方式的学习。

There's this learning of of how the world works.

关于世界如何运转。

Of how the world works.

而关于这个最酷的地方在于，现在你可以设想让系统去想象一系列动作会产生什么结果。

And and the cool thing about this is that now you can imagine you can have the system imagine what would be the outcome of a sequence of actions.

如果你给它一个目标，比如我希望世界最终呈现这种状态，你能找出一系列动作来帮我实现这个目标吗？

And if you give it a goal saying, like, I want the world to look like this at the end, can you figure out a sequence of actions to get me to that point?

它实际上可以通过搜索一系列能产生该结果的动作来解决问题。

It can actually figure out by search for a sequence of actions that will actually produce that result.

这就是规划。

That's planning.

这就是推理。

That's reasoning.

这是真正的推理和实际的规划。

That's actual reasoning and actual planning.

好的。

Okay.

Jan，我得让你说说，虽然我们时间有限，但你能用60秒谈谈对DeepSeek的看法吗？开源模型现在是否已经超越了专有模型？我们真的只能给你60秒。

Jan, I have to get you out here where we are over time, but can you give me, like, sixty seconds your reaction to DeepSeek and sort of has open source overtaken the propriety model proprietary models at this point, and we gotta limit to sixty seconds.

不然你的团队会杀了我的。

Otherwise, I'm gonna get killed by your team here.

所以

So

超越这个词有点重了。

Overtaken is a is a strong word.

我认为开源世界的进展更快。

I think progress is faster in the open source world.

这是肯定的。

That's for sure.

但当然，你知道，专有厂商也在从开源世界的进步中获利，对吧？

But, of course, you know, the propriety shops are profiting from the progress of the open source world, right?

他们和其他人一样都能获取这些信息。

They get access to that information like everybody else.

所以很明显的是，开源世界涌现出更多有趣的创意，任何一家公司无论规模多大都无法独自想到。

So what's clear is that there is many more interesting ideas coming out of the open source world that any single shop as big as it can be cannot come up with.

没有人能垄断好创意。

Nobody has a monopoly on good ideas.

因此开源世界的魔力在于它能汇聚全球的人才。

And so the magic efficiency of the open source world is that it recruits talents from all over the world.

因此我们在DeepSeek项目中看到，如果组建一个小型团队并给予较长研发周期和较少限制条件来开发下一代大语言模型，他们确实能提出前所未有的创新想法，对吧？

And so what we've seen with DeepSeek is that if you set up a small team with a relatively long leash and few constraints on coming up with just the next generation of LLMs, they can actually come up with new ideas that nobody else has come up with, right?

他们能在某种程度上重新定义做事方式

They can sort of reinvent a little bit what how you do things.

如果他们将这些成果与世界共享，整个人类文明都会进步

And then if they share that with the rest of the world, then the entire world progresses.

明白吗？

Okay?

因此这清楚地表明开源发展更快，开源世界能孕育更多创新，而专有领域可能难以追赶。

And so it clearly shows that open source progresses faster, and a lot more innovation can take place in the open source world, which the proprietary world may have a hard time catching up with.

它的运营成本更低。

It's cheaper to run.

我们看到的是，你知道，与我们交流的合作伙伴表示，他们的客户在原型设计阶段可能会使用专有API。

What we see is for, you know, partners who we talk to, they say, well, our clients, when they prototype something, they may use a proprietary API.

但当真正部署产品时，他们实际上会使用Lama或其他开源引擎，因为这样更经济且更安全。

But when it comes time to actually deploy the product, they actually use Lama or open or other open source engines because it's cheaper and it's more secure.

要知道，它更可控。

You know, it's more controllable.

你可以本地部署运行。

You can run it on premise.

明白吗，它有各种优势。

You know, there's all kinds of advantages.

所以我们看到一些人的观念也发生了很大转变，他们最初担心开源成果会...比如说帮助中国之类的，如果你有地缘政治方面的顾虑。

So we've seen also a big evolution in the thinking of some people who are initially worried that open source efforts were going to, I don't know, for example, help the Chinese or something, if you have some geopolitical reason to think it's a bad idea.

但DeepSea表明的事实是，中国人并不需要我们。

But what DeepSea has shown is that the Chinese don't need us.

我是说，他们自己能想出非常棒的点子。

I mean, they can come up with really good ideas.

对吧？

Right?

我们都知道中国有非常非常优秀的科学家。

I mean, we all know that there are really, really good scientists in China.

而鲜为人知的是，整个科学界被引用次数最多的论文是一篇十年前（2015年）关于深度学习的论文，它出自北京。

And one thing that is not widely known is that the single most cited paper in all of science is a paper on deep learning from ten years ago, from 2015, and it came out of Beijing.

哦。

Oh.

好的。

Okay.

这篇论文名为《ResNet》。

It's the paper is called ResNet.

这是一种特殊类型的神经网络架构，其基本设计理念是：深度学习系统的每个阶段默认都计算恒等函数。

So it's a particular type of architecture of neural net where basically by default, every stage in a deep learning system computes the identity function.

你只需将输入原样复制到输出端。

You just copy its input on its output.

而神经网络要做的是计算与这个恒等关系的偏差。

And what the neural net does is compute the deviation from this identity.

这使得训练具有数十层（甚至可能上百层）的超深度神经网络成为可能。

So that allows to train extremely deep neural net with dozens of layers, perhaps 100 layers.

这篇论文的第一作者是一位名叫何恺明的先生。

And he was the first author of that paper is a gentleman called Kamin He.

当时，他在微软亚洲研究院北京分部工作。

At the time, he was working at Microsoft Research Beijing.

嗯。

Mhmm.

在那篇论文发表后不久，他加入了加州FAIR（Facebook人工智能研究院），我聘请了他，他在FAIR工作了大约八年，最近离职，现在是麻省理工学院的教授。

Soon thereafter, the publication of that paper, he joined FARE in California, So I hired him and worked at FAIR for eight years or so and recently left and is now a professor at MIT.

所以世界各地确实都有非常优秀的科学家。

So there are really, really good scientists everywhere in the world.

没有人能垄断好创意。

Nobody has a monopoly on good ideas.

硅谷当然也没有好创意的垄断权。

Certainly Silicon Valley does not have a monopoly on good ideas.

另一个例子是，事实上第一个Lama模型出自巴黎。

Or another example of that is actually the first Lama came out of Paris.

它出自巴黎的Fair Lives团队，一个由12人组成的小团队。

It came out of the Fair Lives in Paris, a small team of 12 people.

因此，你必须利用全球各地多元化的思想、背景和创造力，如果你想推动科学与技术的快速发展，而这正是开源所促成的。

So you have to take advantage of the diversity of ideas, backgrounds, creative juices of the entire world if you want science and technology to progress fast, and that's enabled by by open source.

Jan，和你交谈总是很愉快。

Jan, it is always great to speak with you.

这是我们，我想，第四次或第五次交谈了，我想，这要追溯到九年前了。

This is our, I think, fourth or fifth time speaking again, going back nine years ago.

你总是帮助我看穿所有炒作和喧嚣，真正理解正在发生的事情。

You always helped me see through all the hype and the buzz and actually figure out what's happening.

我相信这对我们的听众和观众来说也会是如此。

I'm sure that's gonna be the case for our listeners and viewers as well.

所以，Yan，非常感谢你的到来。

So, Yan, thank you so much for coming on.

希望我们能很快再次合作。

Hopefully, we do it again soon.

谢谢你，Alex。

Thank you, Alex.

好的，各位。

Alright, everybody.

感谢大家的观看。

Thank you for watching.

我们将在周五回来解析本周的新闻。

We'll be back on Friday to break down the week's news.

在那之前，我们下次在《大科技播客》再见。

Until then, we'll see you next time on Big Technology Podcast.