[NeurIPS最佳论文] 用于自监督强化学习的千层网络 — 王凯文等，普林斯顿大学

欢迎来到Lanespace。

So welcome to Lanespace.

我们基本上是想为那些无法到场的人提供最佳的NeurIPS播客体验。

We are basically trying to provide the best optimal sort of podcast experience of NeurIPS for people who are not here.

恭喜你的论文。

And congrats on your paper.

感觉如何？

How does it feel?

是的。

Yeah.

这非常令人兴奋。

It was very exciting.

是的。

Yeah.

我们昨天有张贴海报，今天将进行口头报告。

We had a poster yesterday, and then today we'll have an oral talk.

你当时是不是被人群围住了？

Were you just, like, mobbed?

哦，哦，人特别多。

Oh, Oh, There was a lot of people.

连续三个小时，一波又一波的人过来，我们根本忙不过来，但是

It's like three hours straight of like, you know, like waves of people to like that we trying to institute, but

我从来没拿过最佳论文奖。

So I've never received the best paper.

你是刚在网站上看到的吗？

Did you just find out on the website?

就像，什么

Like, what

哦，我有一天早上醒来，检查邮件，然后

Oh, I just like woke up one day and like checked my email and then

啊，他们只是觉得他们只是

Ah, They just thought they just like

他们说：‘哦，我刚看到你的邮件。’

They was like, oh, like, that's like, I just saw your email.

哦，你被授予了最佳论文奖。

Oh, you were like, been awarded best paper.

我只是想，也许从评审意见中你也能知道。

I'm like But maybe you know from the reviews as well.

对吧？

Right?

所以我觉得，是的。

So I think Yeah.

我们知道评审意见显示我们表现不错，但表现好和拿到最佳论文还是有区别的。

We know from the reviews that we did well, there's but a difference between, like, doing well on the reviews and getting best paper.

所以那部分，我们其实并不知道。

So that part, we didn't actually know.

嗯。

Yeah.

所以我稍微跳过了一点。

So I I I skipped a little bit.

也许我们可以一个一个来，简单介绍一下你自己，以及你在团队中做了什么。

Maybe we can go sort of one by one and and sort of introduce, you know, who you are and what you did on on on on the team.

我是凯文。

I'm Kevin.

我来自普林斯顿大学，是本科生。

I was an undergrad from from Princeton.

我刚毕业。

I just graduated.

嗯，我想是我主导了这个项目，启动了它，然后我很高兴能和伊尚在通话中合作，还有本也一样。

And, yeah, I guess I led the project, like, started the project, and then well, I was very happy to collaborate with Ishaan in the call and Ben also.

对。

Right.

你们是在同一个研究小组吗？

And were you in, like, the same research group?

你是怎么，呃，你的意思是？

Like, how do you how what's your Yes.

我们是上下文。

We are context.

所以，是的。

So so yeah.

所以我们都是普林斯顿的。

So we're all from Princeton.

是的。

Yeah.

感谢艾伦帮你们安排的。

We're thanks to Alan for booking you guys.

这个项目实际上起源于一次IW研讨会。

So this project actually started from, like, an IW seminar.

就是本教授的独立研究研讨会。

So like like, an independent work research seminar that Ben was teaching.

这实际上是我早期接触机器学习研究的经历之一。

And this was, like, actually, like, like, one of my first experiences in, like, ML research.

所以能获得这样的经验真的很有价值。

So it was really valuable to, like, get that experience.

当时伊莎娜也在这个研讨班里，研究相关的内容，因此我们在研讨班期间合作了很多。

And then Ishana was also in that seminar and working on adjacent things, so we collaborated a lot during that seminar.

后来，这个项目取得了一些非常酷的结果。

And then, yeah, the project turned out to have some pretty cool results.

再后来，哈特也加入了这个项目，他也在做类似的工作，成为了很好的合作伙伴。

And then later on, also, like, Halt working on sort of similar things also joined it on the project and became, like, a good collaboration.

是的。

Yeah.

我不知道你们有没有人想谈谈，关于如何决定这个研究问题的其他方面。

And I I don't know if any of you guys wanna wanna chime in on, like, other elements of coming into, like, deciding on this problem.

我的实验室主要研究深度强化学习。

So it's, like, probably my lab works on deep reinforcement learning.

但历史上，‘深度’意味着两层、三层或四层。

But historically, deep meant, like, two or three or four layers.

不是一千层？

Not 1,000?

当凯文和肖恩提到他们想尝试非常深的网络时，我有点怀疑这会不会成功。

When Kevin and Sean mentioned they wanted to try really deep networks, I was kinda skeptical it was gonna work.

我以前试过。

I've tried this before.

这行不通。

It doesn't work.

其他论文也试过，但根本行不通。

Other papers have tried this before, and it doesn't even work.

所以一开始我就非常非常怀疑。

So I was very, very skeptical starting out.

我不知道当时有没有表达清楚，但这就是我当时的预设，因为

I don't know if I conveyed this at the time, but that was my prior going in because

但你是否认为你的工作是筛选，或者，嘿，伙计们。

But do you do you view your job as, like, screening or, hey, guys.

这可能行不通。

This is probably isn't gonna work.

你应该试试别的想法。

You should try a different idea.

你知道，还是你应该鼓励，即使它看起来

You know, or should you be encouraging even if it's

很蠢？

dumb?

这是在下注。

It's selecting bets.

是的。

Yeah.

而这是一个我愿意下的赌注。

And this was a bet I was willing to make.

是什么让你愿意下这个注？

What what made you willing to make a bet?

这看起来成本相对较低，而且米哈伊尤其在过去一年里开发了基础设施，使运行这些实验变得容易得多。

It seemed relatively low cost, and that we Mihaly, in particular, had spent the past year developing infrastructure to make it a lot easier to run some of these experiments.

之前的先例表明，更深的网络能表现好得多。

And the precedent was deeper networks could do a whole lot better.

就像深度学习革命在过去一天里所展现的那样。

Like, that's what the deep learning revolution has been over the last day.

是的。

Yeah.

我知道。

I know.

为什么我们不再让网络更深了？

Why do we stop making them deeper?

而强化学习就像是一个例外，我们一直使用这些非常浅的网络。

And reinforcement learning was like this one anomaly where we continue to use these really shallow networks.

在我们所研究的从零开始的场景中，这一点尤其明显。

And that's particularly true in the settings that we were looking at where you're starting from scratch.

你从一无所有开始。

You're starting from nothing.

你们还有其他想补充的观点吗？

Any other perspectives you guys wanna chime in with?

我想我应该先概述一下我们的项目。

I guess maybe I should just go over, like, an overview of our project.

好的。

Yes.

好吧。

Okay.

抱歉。

Sorry.

是的。

Yes.

所以，我对我们的项目的看法是，如果你看一下深度学习的领域，你会看到有自然语言处理、语言、视觉，然后是强化学习。

So the way that I kind of view our project is that if you look at the landscape of deep learning, you know, you have NLP, like language, vision, and then RL.

而且，就像本之前提到的，在语言和视觉领域，我们已经逐渐形成了这种大规模网络的范式。

And, like, as Ben kind of alluded to, you know, like, in in language and vision, we've sort of converged to these, like, paradigms of scaling to massive networks.

对吧？

Right?

比如数百亿甚至万亿级别的参数，深度学习从这种趋势中获得了大量进展。

Like, hundreds of billions of parameters, trillions of parameters, and there's been, you know, a lot a lot gained from in deep learning from from that.

对吧？

Right?

但随后，在深度学习和深度强化学习的第三个分支中，情况似乎并非如此。

And then but then it seems like in the third sort of branch of deep learning in deep RL, that has not yet been the case.

当我进入本的课程和研讨会，观察这些网络时，我非常惊讶。

Like, I was very surprised, like, coming into some like, you know, Ben's class and seminar when I was looking at the networks.

为什么你们只是用一个简单的两层MLP来处理这些前沿的、最先进的强化学习算法呢？

Oh, why were you just using, like, a simple, like, two layer MLP for, like, these frontier sort of, you know, state of the art RL algorithms?

所以我非常好奇。

And so I was very curious.

我们能否设计出强化学习算法？

Like, can we design RL algorithms?

我们能否为强化学习制定一套方法，使其能够以类似于语言和视觉领域的方式进行扩展？

Can we sort of put together a recipe for RL that can allow it to scale in potentially, you know, analogous ways that language and vision might scale.

因此，我们知道传统的强化学习，比如基于价值的强化学习，并不能很好地扩展。

And so what we did is that we know that traditional RL, like, let's say, like, value value based RL doesn't really scale.

对吧？

Right?

这一点在文献中已经非常明确了。

This is pretty clear from the literature.

于是我们尝试了一种不同的强化学习方法，称为自监督强化学习：在这种方法中，我们不是学习价值函数，而是学习状态、动作和未来状态的表示，使得同一轨迹上的表示彼此靠近，不同轨迹上的表示彼此远离。

So we tried a different approach of RL called self supervised RL where instead of learning, like, a value function, we're learning representations of states, actions, and future states such that the representations along the same trajectory are pushed together, the representations along different trajectories are pushed apart.

这是一种不同的强化学习方法，使我们能够以自监督的方式进行学习。

And this is just, like, a different approach to RL that allows us to learn in a self supervised manner.

因此，我们可以在没有任何人工设计的奖励信号的情况下解决任务目标。

So there's we can solve task reach goals without any human crafted reward signal.

我们知道自监督学习在深度学习的这些不同领域中是可扩展的，那么自监督强化学习是否也能以类似的方式扩展呢？

And so we know that self supervised learning is scalable in these different areas in deep deep learning, so can self supervised RL scale in similar ways?

当我们最初尝试时，实际上并没有成功。

When we first tried it, it actually didn't work.

我们让网络变得更深了。

Like, we've made the networks deeper.

性能完全下降了。

The performance, like, totally degraded.

但后来我也尝试了其他方法，当时我想到，在我们的文献中还有一些其他工作，比如我们尝试了残差连接，还有其他一些必须加入到这个方法中的架构组件。

But then we also try but then I separately, was like there's also some other work, like, in in our literature, like, we tried, like, residual connections, and then there's other a few other architectural components that we had to put into the recipe.

然后突然有一天，我运行了这个实验，在某个环境中，当深度翻倍时并没有什么效果，但再次翻倍深度并结合这些新组件后，性能在这一环境中突然飙升。

And then all of a sudden, like, one day, like, I ran this experiment, and there was, like, this one environment in which there was, like, like, going from like like, doubling the depth didn't really do anything, but, like, doubling the depth again with these different components suddenly, like, skyrocketed performance in this one environment.

要让这个方法生效非常不容易，因为通常我们会考虑进行超参数优化。

Getting this to work was very nontrivial in the sense that, like, usually, we wanted to think about doing hyperparameter opt optimization.

我们尝试改变a，看看是否有所改善，再尝试改变b，看看是否有效。

We try changing a, see if it makes it better, try changing b, see whether it makes it better.

如果我们只是增加深度，效果反而更差。

And if we just made the depth bigger, it makes it worse.

如果我们只是添加残差连接，也没有改善效果。

If we just add residual connections, it didn't make it better.

真正让这个方法奏效的是凯文和亚山发现的这些因素的组合。

And it was really this combination of factors that Kevin and Yashan figured out that really made this work.

作为这之前的铺垫，我们也尝试了在不同维度上进行扩展。

And as a precursor to that, we also tried scaling along different dimensions.

比如扩大批量大小，扩大网络的宽度，也就是隐藏层的规模。

So scaling the back size, scaling the the width of the network, so the hidden layers.

M效应。

M effect.

没错。

Yep.

是的。

Yeah.

差不多就像盲目增加深度一样。

Pretty much kinda similar to just scaling depth naively.

是的。

Yeah.

当我们开始引入残差连接、层归一化这些特定的架构选择时，性能才出现了显著提升，达到了某些关键深度，性能提升了好几倍。

And then once we started introducing residual connections, layer norm, these specific architectural choices, that's when we saw significant jumps in performance, like these critical depths at which performance multiplies by a pretty huge factor.

正是在这一点上，我们真正注意到，与单纯扩大规模所带来的性能提升相比，我们解锁了显著的性能收益。

And that's where we really noticed, like, unlocking some significant performance gains as opposed to scaling just along with which did yield some performance improvements.

但当你观察随着规模扩大，网络参数数量的变化时，它大致呈二次方增长，而不是像增加深度那样。

But when you look at the number of parameters that your network has as you grow with, it's roughly a quadratic as opposed to something like growing depth.

因此，从我们进行的实验来看，这种方法在某种程度上更具参数效率，也更具样本效率。

So it's more in some sense, it's more parameter efficient, also more sample efficient from the experiments that we conducted.

不错。

Nice.

在某些方面，你实际上是在复制一些在现实世界中出现的现象

In some ways, you're sort of replicating stuff that is seen in

但使用的是一个你可以研究的小型模型。

the wild, but on on a very small model that you can study.

你会不会说这是

Is that would you would you say that's

是的。

Yeah.

我想补充一下凯文之前说的内容。

So I kind add to what Kevin said earlier.

我们看到，通过扩大语言模型和图像生成模型的规模和深度，性能得到了巨大提升，这看起来非常直观。

We saw these huge performance improvements in language models, image generation models by making them larger, making them deeper, which seems very intuitive.

是的。

Yeah.

因此，我们的工作借鉴了基础研究，比如残差网络，它通过残差连接来避免梯度消失，这一点我们在论文的后续部分的一些消融实验中也展示了。

And so that's why our work, we we draw from, like, foundational research, right, like residual networks, which employ residual connections to avoid vanishing gradients, and that's something that we show in some of our ablations in our in our paper, like, further down.

这可能在附录中，我们做了没有这些残差连接的实验。

It's probably in the appendices where we did experiments without these residual connections.

因此，我们借鉴了其他领域已有的概念，并将其应用于强化学习的场景，证明了其有效性。

And so it's sort of borrowing these concepts that have existed in other fields and applying them to this setting with RL and showing that it works.

在本恩离开之前，我让他来做最后的总结。

Before Ben has to has to go, I'll I'll leave the sort of last word to him.

这项工作启发了哪些你接下来想推进的额外研究？

What additional work does this inspire that, like, that that you want to push on next?

我想对论文澄清一点，然后直接回答这个问题。

I think there's one thing I'd clarify about the paper, and then I'll directly answer the question.

好的。

Yes.

我认为可能需要澄清的是，很多读到标题的人会想，哇。

I think the thing I might clarify about the paper is think a lot of people reading the title are like, wow.

大网络。

Big networks.

它们很棒。

They're great.

我选大网络。

I'll take big networks.

你现在解决了。

You solved it now.

我们可以走了。

We can just go.

是的。

Yeah.

我们直接把大网络加到PPO里。

We just take big networks from add them to PPO.

加到SAC里。

Add them to SAC.

加到你最喜欢的强化学习算法里。

Add them to your favorite reinforcement learning algorithm.

但我认为这实际上并不是主要结论。

I But think that's actually not the main conclusion.

我认为主要结论是，使用大网络不仅需要这些架构技巧，而且如凯文之前提到的，还需要使用不同的目标函数。

I think the main conclusion is that using big networks not only requires these architectural tricks, but also, as Kevin mentioned before, it requires using a different objective.

这个目标函数实际上并不使用奖励。

This objective doesn't actually use rewards in it.

因此，标题中的另一个词——强化学习——也可能有点用词不当，因为我们并没有直接试图最大化奖励。

And so there's another word in the title, reinforcement learning, that also might be a little bit of a misnomer because we aren't directly trying to maximize rewards.

我们的代码中没有一行是说‘最大化奖励’的。

Our code doesn't have a line of code saying maximize rewards here.

那么，归根结底，这还是一个强化学习方法吗？

And so is, at the end of the day, this a reinforcement learning method?

我不知道。

I don't know.

它看起来更类似于机器学习其他领域的自监督方法。

It looks much more similar to the self supervised methods in other areas of machine learning.

因此，我认为这种方法和这项工作实际上处于强化学习和自监督学习研究的一个有趣交汇点上。

And so I think that the method and the work really stands in some sort of interesting intersection of reinforcement learning and self supervised learning research.

我们在海报的左下角放了一个小图，那是从一段关于如何构建智能系统、以及这将由无监督学习、监督学习还是强化学习实现的幻灯片中截取的截图。

And we had this little figure on the bottom left of the poster, which was the screenshot of a slide from talking about how to build intelligent systems and whether that's gonna be done by unsupervised learning or supervised learning and reinforcement learning.

我认为她的论文真正表明，这些方法之间的界限非常模糊，而构建智能系统的关键可能在于综合利用它们的所有洞见。

And I think what her paper really suggests is that the boundary between these things is really blurry, and maybe the keys to building intelligent systems are gonna be leveraging insights from all of them.

是的。

Yeah.

这就像一层层的蛋糕。

It's a layer cake.

没错。

Exactly.

谢谢你的宝贵时间。

Well, thank you for your time.

我知道你很快就要走了。

I know you I know you have to go soon.

期待

Looking forward

是啊。

to yeah.

多伦，谢谢。

Doron, thanks.

嗯。

Yeah.

非常感谢你的到来。

Thank you so much for coming.

我觉得那种模糊界限的见解很有趣。

I I think that that insight of, like, blurring things is interesting.

我不确定你是否提到过，比如表征学习的抽象层。

I I don't know if you like, you you were talking about sort of, like, the abstraction layer of representation learning.

我不知道这是否让你想到自监督学习和强化学习之间的融合。

I don't know if if that triggers anything in terms of, like, the mix between self supervised and reinforcement learning.

这是你们发现的一个根本性问题，还是人们读论文时普遍不理解的地方？

Is that is that something fundamental that you've discovered or that we've that people don't understand when they when they read the paper?

是的。

Yeah.

我认为最好的解释方式是，我们知道标准的强化学习并不具备很好的可扩展性。

I think the best way that I would explain it is that we know that standard RL is not super scalable.

那么，为什么这种不同的方法或不同的目标强化学习可以具备可扩展性呢？

And so, like, why can this different approach or different objective RL be scalable?

我认为这是因为我们从根本上将学习的负担从类似Q学习或回归TD误差（我们知道这些方法非常不稳定、嘈杂且有偏差）转移到了一个本质上是分类的问题上。

I think it's because we're fundamentally shifting the burden of learning from something like q like q learning or, like, regressing to, like, TD errors, which we know is quite spurious and noisy and biased, to fundamentally, like, a classification problem.

我们试图分类的是，未来状态是否属于同一条轨迹，还是属于不同的轨迹。

We're trying to classify whether future state is along the same trajectory or along a different trajectory.

而我们是通过表征学习来实现这一点的。

And we do this with representation learning.

对吧？

Right?

我们知道，在深度学习文献中，分类、交叉熵损失和表示学习都是可扩展的。

And we know that classification, cross entry loss, and representation learning is scalable in the deep learning literature.

对吧？

Right?

如果我们想想语言以及其中的一些目标函数。

If we think about language and, like, some of the objectives there.

从某种意义上说，我们正在模糊我们正在进行强化学习的界限。

So in some sense, we're kind of blurring the the lines that we're doing reinforcement learning.

它仍然是一个演员-评论家强化学习算法。

It's still an actor critic reinforcement learning algorithm.

它就像一个目标条件的强化算法。

It's like a goal condition reinforcement algorithm.

但解决这个强化学习任务的学习负担，已经转移到了类似于语言和视觉领域中那些已被证明高度可扩展的目标上。

But the objective the burden of, like, learning of the of of solving that RL task shifts to something that's more similar to objectives that you might see in language and vision that we know have scaled so much.

所以我认为是的。

And so I think yeah.

我认为这是我们看到的一个基本洞见：通过这种不同的方式来处理强化学习，我们能够获得远超传统ARA方法所能做到的成果，我们的网络规模也得到了显著扩展。

I think that's, like, one of the fundamental insights that we've seen is that it seems like by approaching RL in in this different approach, we're able to, like, get so much more out of we were able to scale our networks, like, significantly beyond what was, like, standard used in ARA.

我可以插一句吗？

Can I jump in?

我来再多补充一点关于架构的背景，因为是的，我们使用了另一个目标函数，即对比损失。

I will just give a bit more of context about the architecture because, yeah, we use another objective and the influence is or the contrastive clause.

不过，我们的架构与之前的工作，比如Brau或Simba Fowl、Simba Fowl One、Simba Fowl Two等论文非常相似。

However, the architecture is quite similar to the previous works of previous papers like Brau or Simba, Simba Fowl, Simba Fowl Simba Fowl one, Simba Fowl two.

因此，我们也对这个架构做了一些调整。

So we also tweaked a bit of this architecture.

但并不是说我们从零开始发明了轮子。

However, it's not that we like invented the wheel for the first time.

真正让规模大幅提升、性能随规模增长的，是架构与目标函数的结合。

It's the merging between the architecture and the objective that makes the scale really, like, go up and and performance follow the the scale.

我认为这一点我们可能需要更深入地思考。

I think that's something that we should probably mind deeper.

你觉得，比如说，你已经在哪些领域或行业应用了它？在多种不同的网络或数据集上？

Do you think I guess, like, what domains, what industry like, you've applied it on on multiple different types of networks that are or datasets.

你认为有没有哪个领域是特别容易上手的、低垂的果实？

Is there a particular affinity that you think, like, has is, like, sort of low hanging fruit?

是的。

Yeah.

实际上，如果你看看我们很多任务，特别是机器人任务。

So, actually, if you look at a lot of our tasks there, particularly sort of like robotics tasks.

首先，我非常好奇这类工作会对机器人领域产生怎样的影响。

So this is first thing, I'd be very curious about how a work like this could impact, like, the robotics field.

据我了解，现在的机器人技术有几种不同的方法。

Like, my understanding of robotics is that a lot of robotics are now there's kind of multi a few different approaches.

一种方法是希望通过模仿学习来训练机器人。

Like, one approach is we wanna train robots using imitation learning.

所以我们试图收集海量的数据。

So we try to collect, like, an insane amount of data.

我们有大量的人类监督数据，试图扩展这些数据，并通过模仿学习进行训练。

We have a ton of cubers human supervision, and we try to scale up this data, and we're, like, learning with imitation learning.

但另一方面，可能还有另一种方法，比如目标条件强化学习，我们可以训练机器人代理和强化学习代理，在完全不需要人类监督的情况下解决有意义的任务。

Like but on the other hand, potential like, perhaps there's another approach, which is, like, for example, like, goal condition reinforcement learning, where we can actually train robotic agents and train RL agents to solve meaningful tasks with absolutely no human supervision.

无需演示。

No demonstration.

这更具可扩展性。

It's much more scalable.

是的。

Yeah.

所以，嗯，所以

So, yeah, so

这可以作为一种替代方法。

this could serve as an alternate approach.

也许与其扩展数据、扩展人工人类监督（这并不太容易扩展），不如想办法让目标条件强化学习变得可扩展，比如我们可以扩展架构，或者扩展

And perhaps instead of, like, scaling data, like, scaling manual, like, human supervision, which which is, you know, not super scalable, if there are ways to sort of make goal condition reinforcement learning scalable and, like, we can just scale the architecture or we can scale

因为你专注于所有的目标。

Because you're focused on all your objectives.

是的。

Yeah.

对。

Right.

对于某些不同的目标，我认为这可能会非常令人兴奋，看看它如何影响机器人技术这样的领域。

With with certain different objectives, I think that could be very exciting and see to see how that can affect a field like robotics, for example.

是的。

Yeah.

再深入探讨一下Igor提到的效率问题。

Double click on on just one more one thing on the efficiency, which you Igor was talking about.

我预计越深，情况应该会呈平方级恶化。

I would expect the very deep the deeper it is, this should be quadratically worse.

我不太熟悉现有的文献。

I am not familiar with, like, the the preexisting literature.

我只是在慢慢理清一些直觉。

I'm just, like, sort of working out intuitions.

嗯。

Mhmm.

但总的来说，你发现了哪些权衡因素，我觉得你可能想提醒大家注意？

But, basically, what are the trade offs that you've found that I think you might wanna warn people about?

因为你才是那个提到效率的人。

Because because you you are the guy who mentioned efficiency.

嗨。

So Hi.

当然。

Sure.

当然。

Sure.

当然。

Sure.

是的。

Yeah.

所以我指的是我们海报和论文中的一个图表，其中我们比较了模型在深度和宽度两个维度上扩展时的参数数量，是的。

So I was referring to, like, one of the figures on our poster also in our paper where we compare, like, the number of parameters that models have as we scale along the axis of depth and as we scale along the axis of width Yeah.

从我们的基线架构来看，最基础的版本是宽度为256，即隐藏层有256个神经元，深度为4层，也就是4个隐藏层。

From our baseline architecture the most baseline one would be, like, a width of two fifty six the hidden layers of two fifty six neurons, and then the depth is four four layers or hidden layers.

所以我当时想表达的是，当你在深度维度上扩展时，模型的参数数量大致呈线性增长。

And so the point I was making there is that when you scale along depth, your the number of parameters that your model has is gonna grow roughly linearly.

而当你让网络的输出变宽时，下一个网络的输入也会随之增长。

Whereas with with, you're making your network outputs wider, and then the input to the net next network is also growing as well.

因此，你的网络参数数量将大致呈二次方增长。

And so the the number of parameters your network's then gonna have grows approximately quadratically.

因此，我们做了一个实验，研究在通过这两种不同方式扩展模型参数数量时，对于大致相同的参数数量，哪种方式能带来更好的性能。

And so one of the experiments we did was sort of examining, as we grow the number of parameters in our model by scaling along these two different choices, which one for the same, like, approximate number of parameters yields a better performance.

而深度曲线大致是这样的。

And the depth curve kinda goes like this.

它上升得非常快。

It jumps up pretty fast.

这在我们的论文中随处可见。

That's, like, present throughout our paper.

对于宽度来说，增长得稍微慢一些。

For width, it grows a little bit more slowly.

因此，从中得出的结论是，如果你的资源比较有限，按深度扩展可能更好，因为这样可以用更少的参数、更小的模型、更少的可学习参数——而这些是昂贵的。

And so that the kinda takeaway from that is that if you are a bit more resource constrained, scaling long depth might be better because with fewer parameters with a smaller model tool a smaller number tool learnable parameters Which is expensive.

这是昂贵的。

Which is expensive.

没错。

Exactly.

而且一般来说，参数越多，成本自然也越高。

And in general, of course, like, more parameters is also gonna be more expensive.

所以，使用这些网络时，这也是一个需要考虑的额外因素。

So that's just, like, another consideration to think about when using these networks, I suppose.

是的。

Yeah.

还有其他类似的经验法则可以提取吗？

Any other sort of rules of thumbs like that that I can extract?

这只是我能想到的最基础的一个。

This is just the most basic one that I could think of.

是的。

Yeah.

我不知道还有其他的吗？

I don't know if there's any others?

是的。

Yeah.

我想，回到你最初关于权衡的问题。

I guess, like, to your original question of, like, the trade offs.

其中一个权衡或局限性是，显然，如果你让网络更大，运行时间会更长。

Like, one of the trade offs one of the limitations that we say is, like, obviously, if you make the networks bigger, the it it will takes longer to run.

对吧？

Right?

所以，如果你在某个深度层次上把网络深度翻倍，前向传播的时间可能会增加一倍。

So if you, like, double the depth at some level of depth, you you it might take, like, twice as much to, like, take make a forward pass through the network.

对吧？

Right?

然而，在我们的论文中，对于大多数环境，我们实际上能够在很少的层数内就达到近乎完美的性能，根本不需要达到上千层。

However, this is not so, like, within our paper, like, for most environments, we are able to, like, saturate like, get to, like, almost perfect performance within just you know, we don't even need to get to, like, a thousand layers.

比如，可能只需要64层就足够了。

Like, maybe just 64 layers, for example, is sufficient.

在这个范围内，网络的延迟实际上并不一定构成显著的瓶颈。

And in this regime, like like, the the latency of the network is not necessarily actually even the not necessarily, like, a a significant bottleneck.

你可以想象，在很多任务中，尤其是在强化学习中，收集数据才是真正的瓶颈。

Like, you can imagine there's a lot of tasks in which, especially in RL, that, like, collecting data might be the bottleneck.

对吧？

Right?

通过我们的网络进行前向传播可能并不是瓶颈。

And making forward passes through our network may not be the bottleneck.

因此，在我们的环境和研究中，我们专门使用了基于 JAX 的 GPU 加速环境 JAX GCRL。

And so in our environment, we in our research, we specifically use the JAX g c r l environment, which is a JAX based GPU accelerated environment.

因此，我们可以同时并行收集数千条环境轨迹，这样我们就能轻松实现这一点。

So we can collect, like, thousands of, like, environment trajectories, like, in parallel at the same time so that we're able to, like, make, like Oh, this is built in.

对。

Right.

这是内置功能，使我们能够同时在所有这些环境中收集上千条轨迹。

This is built in so that we can collect, you know, like like, a thousand trajectories at the same time along all these environments.

这确保了我们有足够的数据来充分学习。

And so makes that make sure that, like, we have enough data to, like, saturate the learning from things.

哇。

Wow.

是的。

Yeah.

这就像他们所称的工作。

That's, like, work they've been called.

好的。

Okay.

而且我不知道你是否想深入探讨一下

And you I don't know if you wanna explore expound up

关于DoctorZC时代。

on that on the DoctorZC era.

也许我可以，而且你知道，大多数人对PyTorch比较熟悉，但对JAX可能不太了解。

Maybe I can And and, you know, most people are familiar with PyTorch, maybe less familiar with JAX.

关于JAX。

With JAX.

我认为JAX正在获得关注，尤其是在强化学习领域，因为对于在线强化学习来说，获取尽可能多的数据是最重要的。

I think JAX is getting the the traction, especially in RL field because the in for online reinforcement learning, getting as much data as you can is is the most important.

一定有PyTorch的等价物。

There's gotta be a PyTorch equivalence.

但无论如何，对于其他也在探索这种 rollout 的人，有什么建议吗？

But anyway, how are any tips for other people also exploring this kind of rollout?

是的。

Yeah.

是的。

Yeah.

所以我认为我也可以推荐一下，对于条件良好的强化学习，我用的是 JAX DRL，但也有一些多智能体 JAX 实现和其他方法。

So I think I can also recommend, like, for for well conditioned RL, I'm JAGS DRL but there are also like multi agent JAGS implementation and others.

回到我们的论文，如果你看这些图表，会发现只有当我们跨越五千万次转换的门槛时，性能才出现巨大提升。

So going back to our paper, if you look at the plots, only see this like huge performance increase when we cross like 50 millions of transitions gap.

所以我认为这里的数据至关重要。

So I think the data is crucial like here.

是的。

Yeah.

我想进一步补充一下，我喜欢将这些成功经验与深度学习其他领域的成就进行类比。

I guess even to build on that, like I like drawing analogies to like successes in other areas of deep learning.

比如，在大型语言模型中，我们之所以能够扩展到如此庞大的网络，是因为我们找到了一种范式，可以利用整个互联网规模的数据进行训练。

Like, for example, in large language models, the reason why we're able to scale to such large networks is that we found a paradigm in which we can leverage the entire Internet scale of data to alert.

对吧？

Right?

因此，传统上，强化学习中的数据很难获取。

And so data in RL traditionally has been hard to come by.

但现在有了这些GPU加速的环境，我们只需几小时就能收集到数亿倍的数据。

But now with these, like, GPU accelerated environments, we can collect hundreds of millions of times of the data within just a few hours.

因此，我认为这为我们提供了一个绝佳的测试平台，让我们能够找到扩展网络容量的方法，并获得类似的提升。

And so I think that this serves as, like, a really good test bed for us to be able to also find ways to scale up, like, network capacity and get similar kind of gains.

我想我之前问过，你是不是说你在LLM的预训练上会采取不同的做法？

I think I asked about Are you saying that you have a difference you would do pretraining differently in LLMs?

那么，现在的目标有什么不同？

Like, what's the what's the difference objective now?

嗯。

Yeah.

很简单，你所提到的范式是下一个词，没错。

Very simply very simply, the the paradigm that you're referencing is next word Yeah.

或者下一个标记。

Or next token.

对吧？

Right?

这非常稳健。

It's very robust.

嗯。

Yeah.

嗯。

Yeah.

你如何改变这一点？

How do you change that?

哦，我不是说我们在改变它。

Oh, I'm not saying that we're changing it.

我想利用这些见解来提升士气。

I wanna leverage insights from that to apply to morale.

我觉得他应该走另一条路。

I I feel like he should go the other way.

你觉得他应该走另一条路。

You think he should go the other way.

也许吧。

Maybe.

我的意思是，那也会是一个非常有趣的研究方向。

I mean, that that would be a very interesting research direction too.

但事实上，是的，就这一点而言，我一直在想，我们的目标函数在某些情况下并不是严格的下一个词预测，而更像是下一个状态预测。

But actually, yeah, even on that point, like, one of the things I was thinking about is that the way that our our objective works is in some set it's not exactly next word prediction, but it's kind of like next state prediction.

对吧？

Right?

你可以想象，你处于某个当前状态和某个当前动作，我们想要预测这个未来状态——这个特定的状态——是否属于同一条轨迹的未来状态，还是另一条轨迹的未来状态。

You imagine you're at some current state and you're at some current action, and we wanna predict whether or not this future state this this certain state is a future state along the same trajectory or a different trajectory.

所以某种程度上，我们实际上是在做某种类似

And so in some sense, we are actually doing some sort of, like

隐式世界模型。

Implicit world model.

隐式的，比如，

Implicit, like,

你知道，我在想

you know, like, in in I don't know

如果这是个不好的词，那这就是。

if that's a bad word this this is.

或者，在语言中，你用交叉熵损失来分类下一个词元。

Or, like, in language, you you do a cross entry loss to classify the next token.

对吧？

Right?

而在这里，我们只是对某个下一个状态是否属于同一轨迹进行二分类。

And here, we're just doing a binary classification of, like, whether or not some next state is some Yeah.

是的。

Yeah.

是的。

Yeah.

这是一种分类。

It's a classification.

是的。

Yeah.

是的。

Yeah.

是的。

Yeah.

所以我确实看到这里有一些相似之处，或许我们应该深入探讨一下，到底是什么让深度学习能够扩展，以及我们如何利用这些洞察，如何提炼出这些核心原理，并将其应用到各个不同领域，无论是语言还是强化学习。

And so I do see that there are some, like, sort of parallels here that perhaps we should dig into deeper and see, like, what is the core to of what enables deep learning to scale, and then how can we, like, leverage that how how can distill those, like, insights and then apply those across, like, all different fields, whether it's language or reinforcement learning.

是的。

Yeah.