Dive Club 🤿 - 鲁兹·马德维安 - 在Neuralink与苹果设计前沿界面 封面

鲁兹·马德维安 - 在Neuralink与苹果设计前沿界面

Rooz Mahdavian - Designing frontier interfaces at Neuralink and Apple

本集简介

本期节目我们探讨B2B SaaS领域的工艺设计...但为苹果手表乃至神经接口等前沿界面做设计又是怎样的体验?我们将在本周与Rooz Mahdavian的对话中揭晓答案。(https://x.com/roozm) 这位Neuralink的设计工程师将带我们深入技术内核,聊聊如何设计仅凭意念就能操作电脑的体验。其细节把控与第一性原理设计思维令人叹服。 精彩看点: - 神经接口设计中如何营造愉悦体验 - Neuralink对第二位设计工程师的岗位要求 - 在苹果与Neuralink设计前沿界面的核心要素 - 从苹果手表Siri表盘到脑机接口的演进之路 - Rooz关于"与计算机白日梦"及视觉图像直传的构想 - 神经接口光标的重构——从色彩反馈到环形准星 - 更多深度内容 - 苹果手表表盘设计团队 (https://www.youtube.com/embed/oaqHdULqet0?start=585&end=705&autoplay=1&controls=0) - Siri表盘(实习生作品最终量产)(https://www.youtube.com/embed/oaqHdULqet0?start=585&end=705&autoplay=1&controls=0) - Neuralink脑机接口公司官网 (https://neuralink.com/) - Blindside视觉修复项目 (https://neuralink.com/trials/visual-prosthesis/)

双语字幕

仅展示文本字幕,不包含中文音频;想边听边看,请使用 Bayt 播客 App。

Speaker 0

我们经常讨论B2B SaaS领域的工艺,但设计前沿界面是什么体验?

We talk a lot about craft in b to b SaaS, but what's it like designing frontier interfaces?

Speaker 1

我认为这个世界会出现新的交互模式,它们与我们用物理手部动作来操控的方式将截然不同。

I think there will be new interaction models in this world that are, yeah, just very different than having to use our, like, physical hands to, like, articulate.

Speaker 0

脑机交互的未来如何影响设计实践?

How does the future of brain computer interaction shape the practice of design?

Speaker 1

今天我们称之为氛围编码的技术,在未来,模型生成你所描述产物所需的时间,我认为将缩短至一帧。

What we today call vibe coding, you know, in the future, like, the time that it takes for a model to actually produce an artifact that you described will, I think, come down to a frame.

Speaker 1

这将是一次非常刺激而狂野的体验,你可以坐在电脑前,基本上就是和它一起做白日梦,屏幕上会弹出与你脑海中想法直接延伸相关的内容。

This will be a very exciting and wild experience where you can just sit down on a computer and basically daydream with it, and things will pop up on screen that are a direct sort of extension of what you have in your head.

Speaker 0

欢迎加入潜水俱乐部。

Welcome to Dive Club.

Speaker 0

我叫里德,这里是设计师永不止步的学习之地。

My name is Rid, This is where designers never stop learning.

Speaker 0

本周节目嘉宾是Neuralink的设计工程师鲁兹·马达维安。

This week's episode is with Ruz Madavian, who's the design engineer at Neuralink.

Speaker 0

我们即将深入探讨设计神经界面的体验,这可能会显得相当极客。

We're about to get pretty nerdy and talk about what it's like designing neural interfaces.

Speaker 0

比如,如何让人仅凭意念就能控制光标?

Like, how do you allow someone to control a cursor using just their mind?

Speaker 0

这期节目我最喜欢的部分,是听鲁斯如何从第一性原理出发,思考人机交互的每一个细节。

One of my favorite parts of the episode is hearing how Ruse has had to think through every single detail of human computer interaction from first principles.

Speaker 0

这种层级的设计思维确实令人印象深刻。

The level of design thinking is really impressive.

Speaker 0

不过在深入探讨之前,我们先听听鲁的成长经历——这段旅程其实始于苹果公司。

But before we get into all of that, let's hear a little bit more about Roo's journey because it actually starts at Apple.

Speaker 1

对我而言,这段旅程最早要追溯到高中一年级。

I think the earliest point in that journey for me was freshman year of high school.

Speaker 1

当时伯克利出了一项非常疯狂的研究成果。

There was this, like, research that had come out of Berkeley that was truly wild.

Speaker 1

研究者让人们躺在fMRI机器里观看YouTube视频,收集了海量的实验数据。

They had basically put folks through an fMRI machine, and they had them I think it was watching YouTube videos, and they just collected hours and hours of this data.

Speaker 1

最终他们进行了某种所谓的‘机器学习’,能够根据观看电影时产生的fMRI信号,预测出受试者正在观看的电影画面帧。

And then they ended up doing some form of, like, quote, unquote, machine learning where they could take this fMRI signal that would happen while they were watching these movies and then predict what frame of the movie that they were watching.

Speaker 1

然后他们可以实时将这些画面帧重建,在我们眼中看起来就像是他们思维中的电影。

And then in, like, real time, they could then reconstruct all of these frames into what to our eye looks like then a movie of what they're thinking.

Speaker 1

实验结果非常疯狂,简直超级赛博朋克。

The results were super wild, like super cyberpunk.

Speaker 1

就像是由这些画面帧组成的拼图,你能从中看出一些语义性的内容浮现。

It was like this composition of frames from which you could see these, like, semantic things come out.

Speaker 1

比如你能看到,哦,那里会出现一只鸟,或者是一片海滩。

So you could see, like, oh, there would be, like, a bird or it would be, like, you know, a beach.

Speaker 1

你能从这些噪点中看到一些形状逐渐显现。

Like, you could see some shapes emerge from this noise.

Speaker 1

他们还做了更疯狂的尝试——让受试者在fMRI舱里小憩,然后你就能实际看到他们梦见了什么。

And then they took this extra crazy step of, like, them having them go, like, take a nap inside of one of these fMRI pods, And then you could see in effect, like, what they were dreaming.

Speaker 1

这就像是一旦见识过,就难以忘怀的东西。

This was, like, something that once I saw, was, like, hard to, like, let go of.

Speaker 1

这简直难以置信。

It was just, like, unbelievable.

Speaker 1

在那之前,我其实花了很多时间制作电影。

Up to that point, I'd spent a lot of time actually making movies.

Speaker 1

所以你能将脑海中的画面直接展示给别人这个想法,对我来说真的非常神奇。

So the idea that you have these images in your mind that you can now show other people directly was, like, really, really magical to me for that reason.

Speaker 1

就像你花所有时间试图重现这个过程,当你经历制作短片的过程时。

It's like you spend all of this time trying to recreate that when you go through the process of, like, making a short film.

Speaker 1

我能直接向别人展示这个想法,简直不可思议。

The idea that I could just show somebody that directly was just, you know, incredible.

Speaker 1

那大概是我第一次接触到某种神经接口的概念。

That was, like, I guess, my first exposure to the idea of some form of a of a neural interface.

Speaker 1

另外,我觉得计算机本身就是一种神奇的事物。

And then separately, you know, I was also I think computers in general are a magical thing.

Speaker 1

所以我当时真的深陷其中无法自拔。

So I was I was really going down that rabbit hole.

Speaker 1

这是另一种方式,你可以将脑海中的想法展示给别人,为他人创造一种体验。

That's another way in which you can take something that is in your mind and then show it to somebody else and create an experience for somebody else.

Speaker 1

于是五年后,我在大学里得到了一个疯狂的机会,基本上是在Apple Watch团队的表盘组实习。

So fast forward five years, I had this crazy opportunity in college to, basically intern on the Apple Watch team, the Faces team.

Speaker 1

那时距离Apple Watch发布才一年。

This was like one year after the Watch had come out.

Speaker 1

所以那确实是探索Watch界面可能形态的早期阶段。

So it was really the early days of what the interface for Watch might look like.

Speaker 1

表盘团队也是一个超级棒的工作环境。

And the faces team is also a super amazing place to be.

Speaker 1

因为你实际看到的是表盘本身,而不是深埋在界面层级里的东西。

That is like the thing you actually see all of the time as opposed to being sort of more buried into the layers of the interface.

Speaker 1

我的实习项目后来发展成了Siri表盘。

My interim project was what would go on to become the Siri Watch Face.

Speaker 1

所以我当时对'主动式电脑'这个概念感到非常兴奋——那种时刻陪伴着你的设备。

So I was just, yeah, generally really excited about what a proactive computer might feel like, something that is with you all the time.

Speaker 1

我们有个'随手看表'的概念,某种程度上这是手表的核心价值主张——你抬手看时间。

We have this notion of drive by on watch, which was you're checking the time that is in some ways the fundamental value proposition of a watch is that you go check the time.

Speaker 1

但现在有了新可能:在你查看时间的同时,还能呈现其他信息。

But now there's this, like, kind of opportunity to surface something else while you're checking the time.

Speaker 1

所以从某种程度来说,计算机不再是你主动使用的工具,而是逐渐隐入背景,在你真实世界活动中默默为你提供有用服务。

So the computer in a way is not something you're actively using anymore, but something that sort of fades into the background and can hopefully just do useful things for you while you're out and about in the real the world.

Speaker 1

初代Siri表盘正是瞄准这个方向设计的,不过当时采用的机器学习技术比现在要原始得多。

The first version of the Siri Watch phase was sort of aimed at exactly that, which is, like, again, more primitive forms of machine learning than I think we have today.

Speaker 1

但这种方式让我们能够尝试找出最相关的信息提供给你,无论是即将到来的日历事件这类略显老套的例子,还是更具情境化的内容——比如当你正在散步时,我们可以弹出开始锻炼的选项。

But a way for us to try and figure out what is the most relevant piece of information that we can give you, whether it is like a calendar event that you have coming up, the sort of cheesy examples, but also more contextual things like, you know, if you have been going for a walk, like, we can surface the ability to start a workout.

Speaker 1

对吧?

Right?

Speaker 1

在苹果这样的公司经历整个流程真的非常有趣。

That was just a ton of fun to go through that whole flow at a company like Apple.

Speaker 1

我想,这大概是我第一次体验从头脑中的概念出发,经历初步探索、感受原型设计、实际构建,到最后产品落地的完整过程。

I guess, like, my first experience going end to end from just a concept that you have in your head to the initial explorations of what that might feel like to the actual prototyping work, building it, and then living on that.

Speaker 1

你需要不断重复这个过程,最终才能交付一个让人惊艳的产品。

And you sort of go through this loop over and over again ultimately to ship something that feels awesome.

Speaker 1

所以我回来又干了一年,进一步推进这个项目。

So I came back for another year to build that out further.

Speaker 1

毕业后我正式加入团队,继续从事这项工作。

And then I joined full time after graduating to just sort of continue that work.

Speaker 1

在那里工作一年后,我仍然惦记着高中时初次接触神经接口技术的经历。

So after a year there, I was still thinking about that initial sort of exposure to neural interfaces in high school was definitely like in the back of my mind still.

Speaker 1

当然,Neuralink几年前就已经公开宣布,并且一直在推进这方面的工作。

And of course, Neuralink had publicly announced a couple years prior and was still doing a lot of this work.

Speaker 1

所以在他们2019年7月的首次演示后,我观看了整个过程。

So after their first demo, this was like July 2019, it was just watching that.

Speaker 1

很明显这将成为人们真正能用上的技术。

It was clear that this is actually going to be a real thing that people get to use.

Speaker 1

大概在2019年底,我从可穿戴设备领域转到了神经接口这个新前沿。

It was, like, late twenty nineteen that I made the jump from one Frontier interface, on Watch to another, in this case, neural interfaces.

Speaker 0

简短插一句,然后我们可以继续。

Real quick message, and then we can jump back into it.

Speaker 0

前几天我看到一条让人忍不住停下滚动浏览的推文。

I saw a scroll stopping tweet the other day.

Speaker 0

Tailwind的创作者们直接在Paper上训练输出结果以达到完美。

The creators of Tailwind are working directly on paper to train the output to be perfect.

Speaker 0

他们甚至投资了这家公司。

They even invested in the company.

Speaker 0

所以请花一秒钟想想其中的可能性。

So just think about the possibilities for a second.

Speaker 0

未来你可以先在Paper上设计,然后只需右键点击就能复制完美的Tailwind代码,就像创作者亲手编写的一样。

In the future, you could design something in paper and then just right click and copy the perfect tailwind as if the creators themselves wrote it by hand.

Speaker 0

或者你也可以将现有代码组件导入Paper,直接在画布上进行编辑。

Or maybe you take an existing code component and import it into Paper to make edits directly on the canvas.

Speaker 0

我的意思是,这将彻底改变我们为网页设计和交付用户界面的方式,这也是我重金押注Paper将成为下一代伟大设计工具的又一个原因。

I mean, this is going to totally change how we design and deliver UIs for the web, and it's just another reason why I'm betting big on Paper as the next great design tool.

Speaker 0

你今天就可以试试看。

You can try it out today.

Speaker 0

直接访问dive.club/paper即可。

Just head to dive.club/paper.

Speaker 0

到现在你应该知道,Jitter多年来一直是我制作动画的首选工具,但他们仍在疯狂推出新功能。

By now, you know that Jitter has been my go to tool for animation for years now, but they're still shipping like crazy.

Speaker 0

光是今年夏天,他们就发布了评论功能、钢笔工具、变形动画、文字渐变、Google字体库等一系列更新。

I mean, just this summer, they've released comments, pen tool, morphing, text gradients, Google fonts, and a bunch more.

Speaker 0

如果你还没试过,我保证你会惊讶于用Jitter让设计动起来竟然如此简单。

So if you haven't yet, I promise you will be shocked at just how easy it is to bring your designs to life in Jitter.

Speaker 0

所以今天就动手试试看吧。

So go ahead and give it a try today.

Speaker 0

直接访问dive.club/jitter即可。

Just head to dive.club/jitter.

Speaker 0

过去十五年我每天都在设计产品,但最近半年一切都变了。

I've been designing products every day for the last fifteen years, but in the last six months, everything has changed.

Speaker 0

有了AI的加入,我的创意产出速度比以往任何时候都快,但如果得不到让团队达成一致所需的反馈,这一切都毫无意义。

With AI in the mix, I'm cranking out ideas faster than ever, but none of that matters if I can't get the feedback that I need to get the team aligned.

Speaker 0

而目前,获取异步反馈仍然相当糟糕。

And right now, getting async feedback still kinda sucks.

Speaker 0

所以我正在打造我一直想要的产品,它叫In Flight。

So I'm building the product I've always wanted, and it's called In Flight.

Speaker 0

我每天都用它来分享想法并获取团队反馈。

I use it every day to share ideas and get feedback from the team.

Speaker 0

它彻底改变了我的工作方式,所以我迫不及待想展示给你看。

And it's totally changing the way that I work, so I'm excited to show you.

Speaker 0

目前仅限Dive Club听众开放访问,请前往dive.club/inflight抢占名额。

Right now, I'm only giving access to Dive Club listeners, so head to dive.club/inflight to claim your spot.

Speaker 0

好的。

Okay.

Speaker 0

现在进入正片环节。

Now onto the episode.

Speaker 0

我特别喜欢‘前沿界面’这个说法,因为在LinkedIn上作为一项职位描述,我们有过类似的职称,但你实际上在开发完全不同类型的产品和用户体验,从上到下每个部分都是独一无二的。

I love the phrase frontier interface because on LinkedIn as a line item, you know, we've had similar job titles and yet you're working on fundamentally different types of products and user experiences and, like, every piece top to bottom is unique.

Speaker 1

我认为所谓的前沿界面,就是你完全不知道它的实际交互模式会是什么样子的界面。

I think what that really means to be like a frontier interface is one that you have no idea what the actual interaction model is gonna look like.

Speaker 1

历史上,所谓的计算机的交互模式最终决定了你能用它做什么。

Historically, like the interaction model for the quote unquote computer is what ends up defining all of the things that you can do with it.

Speaker 1

回顾过去六十年,那种交互模式通常只是输入机制的功能体现。

Looking back like sixty years, that interaction model is usually a function of just what the input mechanism is.

Speaker 1

你知道,最初是光笔吗?就像只是屏幕上的某个点?

And, you know, is it the light pen originally, like just some point on the screen?

Speaker 1

还是光标?就是那个加上点击能力和若干次小点击?

Is it the cursor, which is like, you know, that plus the ability to click and some number of little clicks?

Speaker 1

或者是像iPhone那样的直接触控?

Or is it direct touch like the iPhone?

Speaker 1

就像,你知道的,同时用十根手指追踪所有这十个接触点。

So like, you know, all 10 of your fingers at the same time tracking like all 10 of those points.

Speaker 1

这最终决定了计算机能力的边界,因为你表达意图的方式、频率以及精度将在很大程度上定义你能用计算机实现什么功能。

That ends up really defining the shape of what the computer can do because how you express your intent and at what cadence and like resolution you can express your intent is going to define like what you can do with the computer in many ways.

Speaker 1

我们或许可以超越纯粹的物理层面。

We can go potentially beyond just the physical.

Speaker 1

我们现在脑海中已有这个画面,如何捕捉这种意图?

We can capture this intent that right now we have this image in our mind.

Speaker 1

我们该如何立即重现这个画面?

How do we go about recreating that right now?

Speaker 1

我们将其分解为成千上万个动作意图——通过移动双手在屏幕上以某种遮挡程度精确地为像素着色。

We like decompose that into thousands, tens of thousands of motor intents where we move our hands to effectively color a pixel on the screen at some level of obstruction.

Speaker 1

我们非常手动地经历这个过程,直到最终能看到与最初构想相近的形态。

And we go through that process very manually up until finally, we can then see some form of what originally was in our minds.

Speaker 1

如果拥有神经接口,你就不必如此费力——就像眺望远方的地平线,因为意图存在于比我的手部动作更高阶的层面。

If you have a neural interface, you don't have to do that necessarily, just like looking far down the horizon because that intent is there much further up the stack than my hand is.

Speaker 1

你或许可以直接读取这种意图。

You could potentially read that directly.

Speaker 1

这就是我所说的前沿领域——我们确实不知道交互模式会是什么样子,但这种设备本身有可能改变这种模式。

So that's what I mean when I say frontier is we really don't know what the interaction model could look like, but the the sort of device itself can potentially change that model.

Speaker 1

所以关键是要探索整个创意空间。

So it's just about exploring that whole space of ideas.

Speaker 0

那么稍微快进一下。

So fast forward a little bit then.

Speaker 0

是什么让你意识到有机会转向更传统的设计角色?

What was the point where you realized that there was an opportunity for you to step into more of a traditional design role?

Speaker 1

我开始大量制作原型,试图体验第一次下载类似Neuralink的应用、连接植入设备并进行校准的感觉。

I started doing just a ton of prototyping on what it would feel like to, you know, download an app like a Neuralink app and connect to your implant and sort of calibrate them all for the first time.

Speaker 1

所以最早的阶段就是尝试感受人类体验可能是什么样子,以及这是否是我们都真正向往的北极星。

So, like, the the earliest steps in here was just, like, getting some feel for, yeah, what a human experience could be and if this is, like, a north star that we're all really, really stoked about.

Speaker 1

我们花了大量时间纯粹在iOS上做这些,因为那是当时的重点。

Spent a lot of time doing that purely on iOS because that was the focus at the time.

Speaker 1

感觉又回到了苹果的早期岁月,就像面对一张真正空白的画布。

It felt like there was early days at Apple again where it was like you had just a truly a blank canvas.

Speaker 1

然后你就在脑海中不断循环着最初的那个概念。

Then you're just going through the loop over and over again of like some concept that you have initially in your mind.

Speaker 1

接着,你会对这个概念进行十次不同的迭代。

Then, you know, 10 different iterations on that thing.

Speaker 1

最终得到一个可以拿在手里把玩的东西。

And then ultimately something that you can hold in your hand and play with.

Speaker 1

这真的很酷。

So that was really cool.

Speaker 1

但另一方面,由于这个过程太早期,几乎没有什么限制条件能让我们做出真正有意义的东西。

The flip side is though that because it was so early in the process, there were almost like too few constraints to actually do something really meaningful here.

Speaker 1

就像你可以做很多很酷的事情,但最终还是在交互模式上做了太多猜测。

It was like you could do a lot of cool things, but ultimately, like, you're guessing a little bit too much on what the interaction model is gonna be.

Speaker 1

显然,这不是我们为自己设计的东西。

And obviously, it is something that we're not designing ourselves.

Speaker 1

我们是为那些首批参与者设计的,他们将会是某种运动功能障碍患者。

We're designing for, you know, whoever that first participant is gonna be, but it's going to be somebody with some sort of motor disability.

Speaker 1

所以他们的感官体验会与我们截然不同,如果你无法切身感受,就很难真正共情并为此优化。

So it's like they're gonna have a very different sensory experience than we do, so it's really hard to actually try and directly empathize with that and actually optimize for that if you can't feel it.

Speaker 0

我能插一句吗?

Can I drill in Yeah?

Speaker 0

就这一点稍作讨论?

On that point for a second?

Speaker 0

因为你刚才说的让我想到,你们在进行所有这些不同尝试时——

Because that was something that was coming to mind while you were talking where you're doing all these different expirations.

Speaker 0

对。

Yeah.

Speaker 0

可能现在说有点早,但最让我着迷的是,我们设计师常谈共情,但这完全是另一个层面的挑战。

Maybe it's a little bit early, but the part that's so fascinating to me is, like, we talk a lot about empathy as designers, but this is a whole other level.

Speaker 0

明白吗?

You know?

Speaker 0

这确实非常困难。

It's, like, very difficult.

Speaker 0

你不仅仅是在想象一份你从未从事过的工作。

You're not just imagining a a job that you've never had before.

Speaker 0

你是在想象一种你从未真正接近过的存在状态。

You're imagining a state of being that you've never really come close to before.

Speaker 0

那会是什么感觉?

What's that like?

Speaker 1

我甚至可以说,这不是你能想象得足够好以至于能派上用场的事情。

I would go so far as to say that it is not something that you can imagine well enough for that to be useful.

Speaker 1

我们有自己的内部模型,比如想象静止不动时的感受。

We have our own, I think, internal models of, like, what it would feel like to, yeah, basically just try and imagine something without moving.

Speaker 1

这在某种程度上是有效的,比如在进行校准任务时,你确实只需要想象移动。

That works well in the sense that you can just sit there obviously for something like a calibration task when you're actually just trying to imagine moving.

Speaker 1

你可以完成这个步骤。

You can do that step of it.

Speaker 1

而且我认为你能很好地完成这个步骤,因为你不需要实际移动。

And I think you can do that step of it well because you don't have to move.

Speaker 1

所以你只是试着在视觉上聚焦屏幕上的某个东西,感受一下——当你在屏幕上观看某些内容时,你的脑海里是否会产生某种反应,那种感觉几乎像是镜像神经元在作用?

So you just kind of visually try and focus on something on the screen and get a feel for, is there something happening in your mind when you're watching something on the screen where it feels like almost like mirror neurons?

Speaker 1

我用这个词可能不太严谨,但当镜像神经元激活时,你会感觉自己和那个事物之间产生了某种联系。

You know, I use that word loosely, but mirror neurons are firing and you feel like there's a connection between you and that thing.

Speaker 1

这个环节绝对可以想象,因为你能亲身体验到,对吧?

That step of it, definitely can, I think, imagine because you can experience that?

Speaker 1

我们称之为开环系统。

That's what we call open loop.

Speaker 1

这意味着,如果你想象这样一个卡通循环:里面有个人类,人类大脑中产生某种意图。

So what that means is that, like, if you imagine this, like, cartoon loop of, like, there's a human in there, the human has some intent in their mind.

Speaker 1

这个意图转化为具体行动,然后他们观察行动的结果,这些反馈又回到大脑中。

That intent goes down into some action, and then they observe the outcome of that action, and then that goes back into their brain.

Speaker 1

就像存在这样一个小的控制循环。

So it's like there's this little loop control.

Speaker 1

开环就是指这个循环没有闭合。

Open loop just means the loop is open.

Speaker 1

他们在观察,但实际上在屏幕上没有看到任何结果。

They are observing, but they don't actually see any outcome on the screen.

Speaker 1

他们纯粹只是在观察发生的事情。

They're just purely observing what's going on.

Speaker 1

闭环是指你能够实际看到结果,然后这显然会反馈回来并闭合循环。

Closed loop is when you can actually see an outcome, and then that obviously feeds back and closes the loop.

Speaker 1

在开环阶段,你完全可以利用自己的经验和直觉来创造一种感觉适合你所设计任务的体验。

The open loop stage, you can definitely sort of use your own experience and your own instincts to sort of create an experience that feels right for the task that you're designing for.

Speaker 1

但闭环极其困难,因为你最终无法直接体验神经接口。

But closed loop is extremely hard because you ultimately cannot experience the neural interface directly.

Speaker 1

例如,我们有多种方法来模拟这类控制。

So for example, we have various ways of simulating these kinds of control.

Speaker 1

当我移动光标时,我们有一个模型会读取该移动的速度,然后将其转换为某种脉冲表示。

When I move the cursor, we have a model that will basically read the velocity of that movement and then convert it into some spike representation.

Speaker 1

就像这种运动可能的神经表征会是什么样子。

So like what the neural representation could look like for this kind of movement.

Speaker 1

但如果你真的用它来优化界面的某些方面,那会失败,因为你是在移动一个真实的光标。

But if you're actually using that to optimize some aspect of the interface, it's gonna fall flat because you're moving a real cursor.

Speaker 1

所以你有所有这些额外的反馈机制,从你手在空间中的位置,到你移动时感受到的摩擦感,比如

So you have all of these additional feedback mechanisms from where your hand is in space to the feeling of the friction that you feel when you move like

Speaker 0

物理鼠标或

a physical mouse or the

Speaker 1

触控板上的摩擦感,再到点击时的压力。

friction on like a trackpad to the pressure of like a click.

Speaker 1

你根本无法真正体会到那种使用感受,我认为这实际上代表了那些没有这些功能的人仅通过观察屏幕时的体验。

You're not gonna get any real sense of how that feels to use that is, I think, actually representative of what it would feel like for somebody who doesn't have any of those things, and it's just looking at what's on screen.

Speaker 1

所以闭环方面的事情,回答你最初的问题,我认为确实不是你能直接感同身受的。

So the closed loop side of things is definitely, to answer your original question, I think not something that you can empathize with directly.

Speaker 1

我想你可以偶尔尝试体会,但这永远无法成为那种感受的可靠指标。

And I think you can weekly try, but it's never something that is a reliable indicator of what that feels like.

Speaker 1

但即使要达到那种程度,我们也知道他们需要首次连接植入设备。

But to even get there, we knew that they would have to connect to an implant for the first time.

Speaker 1

除此之外,我们知道他们需要校准模型,所以我们需要为开环部分准备一些东西。

And beyond that, we knew that they would have to calibrate the model, so we would need something for the open loop side.

Speaker 1

我们需要基于学术界和其他研究工作的成果,对闭环体验应该是什么样做出最佳推测。

And we would need to have some best guess based on academia and also, like, other work that had been done in research as to what that closed loop thing should feel like.

Speaker 1

我们还希望他们能在实验室之外、在与我们的会话之外实际使用这个系统。

And then we also knew that we wanted them to actually use this, like, outside of a lab, outside of just any sessions with us.

Speaker 1

真正的魔法将发生在室外——当我们不在场时,他们只需用脑机接口就能完成各种操作。

Like, the real magic is all gonna happen outside when we're, like, not in the room and they're just using their their BCI to do stuff.

Speaker 1

我们希望这能成为他们真正可以与之共存的工具。

We wanted this to be something that they could actually just, like, live with.

Speaker 1

所有努力都旨在打造一个可靠的系统,其界面足够简单,让他们能在Mac上直接使用,像普通人一样点击操作、使用电脑。

And all of the work that goes into actually making something that's, like, reliable, has a simple enough interface that they can actually use on literally, like, their Mac to just, like, go around, click on stuff, do stuff with their computer again.

Speaker 1

所以现在我们有了非常清晰的方向。

So we had a really good sense now.

Speaker 1

这已经不再是一个单纯的iOS项目了。

It's not really gonna be an iOS thing anymore.

Speaker 1

它将变成一个纯粹的Mac应用,让他们能重新使用电脑,并构建这种体验的基础模块。

It's gonna be just like a Mac app that enables them to use their computer again and what the basic building blocks were gonna be of that experience.

Speaker 0

我想深入探讨其中一些基础模块。

Wanna drill into some of those building blocks.

Speaker 0

或许我们首先要讨论的就是光标。

And maybe the first one we could talk about is just the cursor.

Speaker 0

比如,作为这个体验的设计者,你需要考虑哪些方面?

Like, what are all of the things that you have to think through as someone that's designing this experience?

Speaker 0

因为我的假设是,有太多我们认为理所当然的东西,传统的B2B SaaS设计师可以直接拿来就用。

Because my assumption is there's so many things that we take for granted, and traditional, you know, b to b SaaS designers just get out of the box.

Speaker 1

我认为光标最终是这种体验的核心焦点。

I think the cursor is the focal point of that experience ultimately.

Speaker 1

是的。

Yeah.

Speaker 1

就像当我们使用光标时,它就是你在二维空间中意图的焦点所在。

It's like, when we use a cursor, it is the focal point of your intent in, you know, two dimensions.

Speaker 1

所以它包含了很多东西。

So it encapsulates, like, a lot.

Speaker 1

我认为B2B从业者以及任何为光标设计界面的人都能免费获得的是,使用光标时融入的丰富感官体验,这些体验如今已如此潜意识化,以至于你根本不会去思考它。

The thing that I think the the b two b people get for free and just really anybody that's building an interface for a cursor gets for free is that there's such a rich spectrum of sensory experience that just goes into using a cursor that I think at this point is so subconscious that you don't really think about it.

Speaker 1

但所有这些额外的感官体验,就是我之前提到的那些。

But it's all of those additional senses that I talked about earlier around.

Speaker 1

比如,你能感受到摩擦感。

Like, you get friction.

Speaker 1

你能感受到压力。

You get pressure.

Speaker 1

你能听到声音。

You get sound.

Speaker 1

这一切意味着拥有一个感觉像是自身延伸的光标,让你只专注于光标的动作。

What all of that means is having, like, a cursor that just feels like a extension of yourself and, like, feels like you're just focused on what the cursor's doing.

Speaker 1

你不再需要过多思考双手的操作。

You don't really think too much about what your hands are doing anymore.

Speaker 1

你在实际开发应用时根本不需要考虑这些事情。

You don't have to think about any of these things when you're actually building an app.

Speaker 1

就像,你知道的,往屏幕上放个按钮那么简单。

Like, that's just like, you know, you put a button on screen.

Speaker 1

你可以为按钮添加某种深度效果,做些精巧的设计让按钮点击感很棒。

You can add some sort of depth to that button and, know, do the really nice things you do to make a a button click feel great.

Speaker 1

但最终当你这样做时,其实已经综合考量了所有这些感官媒介。

But ultimately when you do that, it's taking into account all of these other sensory mediums.

Speaker 1

所以我们的挑战在于:参与者将无法获得这些感官反馈。

So our challenge was like, participants are not gonna have that.

Speaker 1

我们仍然希望光标成为体验的核心,并保持绝佳的操作手感。

We still want the cursor because it is the focal point of the experience to feel amazing.

Speaker 1

那么该如何为光标设计类似的反馈机制呢?

So then what's the right way to bring some form of that feedback to the cursor?

Speaker 1

实现方式有无数种可能。

There's a million ways to do this.

Speaker 1

我们还有上百万种方法没来得及实际制作原型或尝试。

There's a million ways to do this also that we still have not actually even prototyped or tried.

Speaker 1

但我们最初的想法其实并不是现在这个版本的游标。

But our first take on this is actually not the the version of the cursor we have today.

Speaker 1

我们最初尝试的是某种常规可见的游标形式。

But our first take on this was some form of a regular cursor that you see.

Speaker 1

当你连接到植入设备时,它会进行这种狂野的过渡转换。

When you connect to your implant, it does this wild, like, transition.

Speaker 1

它会接管你的macOS游标,然后旋转变成这个全新的、神奇的脑机接口游标。

So it'll, take over your macOS cursor and it'll, like, spin into, like, this new, you know, magical BCI cursor.

Speaker 1

当时核心问题是:我们虽然有了模型——这个稍后再谈——它能把某些意图转化为点击概率。

And then the main question was, yeah, we have the like we have a model.

Speaker 1

所以会出现左键点击概率和右键点击概率的情况。

We'll talk about that later that like will take some intents into a probability of a click.

Speaker 1

(因此)会有左键点击的某种概率,和右键点击的某种概率。

So there'll be some probability of a left click, some probability of a right click.

Speaker 1

移动操作没问题

The moving is fine.

Speaker 1

就像移动光标时,这是你能看到的,但关键在于移动和点击的组合

Like when you're moving a cursor around, that is something that you can see, but it's the composition of moving and clicking.

Speaker 1

如何让这种操作感觉自然?

How do you make that feel?

Speaker 1

不要让它像是一个离散事件

Not like a discrete event.

Speaker 1

就像点击会自然发生一样,这种交互

So like, you know, the click will just happen, interaction.

Speaker 1

顺便说一句,之所以希望它感觉连续,是因为现阶段点击确实存在一些延迟

The reason you want that to feel continuous by the way, is just because there is right now, just at the current stage, there is some latency to that click.

Speaker 1

我们的模型需要一定时间来提升判断信心——确认用户(参与者)确实想在这里点击

It takes some amount of time for our model to ramp up its confidence that, oh yeah, the user is actually, the participant is trying to click here.

Speaker 1

这意味着如果模型是完美的,每次的延迟时间应该是相同的

And what that means is that if the model were perfect, it would be the same amount of latency every time.

Speaker 1

实际上延迟时间就是,你知道的,我们取决于我们植入体的采样频率,目前大约是十五毫秒。

And actually the latency would would just be, you know, whatever the sampling frequency of our implant is, which is like, let's say, fifteen milliseconds right now.

Speaker 1

所以每十五毫秒,电脑就会获取一些关于你神经活动的新信息。

So every fifteen milliseconds, the computer will get some new information about your neural activity.

Speaker 1

在理想情况下,点击动作会在你想到点击的瞬间就发生。

In a perfect world, right, that click would just happen the second you think about the click.

Speaker 1

所以在十五毫秒内完成,这样你其实就不需要用户界面了,因为模型总是正确的,延迟基本为零。

So like within fifteen milliseconds, and then you actually wouldn't really need a UI here because it's the model's always right and the latency is basically zero.

Speaker 1

点击动作将在屏幕上发生。

The click will happen on the screen.

Speaker 1

挑战在于存在一定的延迟。

The challenge is that there is some latency.

Speaker 1

假设延迟大约在100到300毫秒之间。

Let's say it's like a hundred milliseconds, 200, 300, somewhere in that range.

Speaker 1

模型分配的概率有时需要100毫秒才能达到正确的概率阈值。

That probability that it assigns is sometimes it'll be a hundred milliseconds before you get to the right probability.

Speaker 1

有时可能需要300毫秒。

Sometimes it'll be 300.

Speaker 1

特别是当我们思考脑机接口旅程的第一个月会是什么样子时,有很大概率系统有时会出错。

And especially when we're thinking about what the first month of, like, the BCI journey would be like, there's also a very good chance that sometimes it'll just be wrong.

Speaker 1

比如,它会在你无意点击时触发点击。

So, like, it'll click when you didn't intend to click.

Speaker 1

这显然是我们想要解决的问题,但我们希望至少能让这种体验可见,更重要的是可预测。

That's obviously something we wanna solve for, but we want the experience of that to be at least visible and much more importantly to be predictable.

Speaker 1

所以当你初次尝试点击操作时,这不是一个随机事件——经过一段时间后屏幕上才会出现点击,而是一个连续的交互过程,就像用手指按压触控板那样。

So when you are in this flow of, like, trying to click for the first time, that isn't this random thing that just after some amount of time, the click will happen on the screen, but it's this continuous interaction much like when you press your finger down on a trackpad.

Speaker 1

这里存在着连续的交互。

There's a continuous interaction there.

Speaker 1

你可以看到这种交互强度随时间逐渐增强。

And you can see the intensity of that ramp up over time.

Speaker 1

我们的直觉是,这种视觉反馈会比没有任何反馈的突然点击体验要好得多。

And our intuition was that that would just feel a lot better visually than just like the click happening would with no feedback at all.

Speaker 1

所以第一个版本使用颜色来执行这个点击操作。

So the first version of this used color to perform that click.

Speaker 1

我们设计了,我觉得,这种非常漂亮的蓝色代表左键点击,橙色代表右键点击。

We had, I think, this really nice blue for a left click and this orange for right click.

Speaker 1

我们会根据每种点击的相对概率来混合这两种颜色。

And we would actually mix the two depending on the relative probability of each click.

Speaker 1

同时我们还利用了深度和光标轻微的透视变化。

And then we also use depth and a slight perspective shift on the cursor.

Speaker 1

当你在点击时,光标会稍微向内倾斜一点。

So as you were clicking, the cursor would sort of tilt inward just a little bit.

Speaker 1

倾斜程度由左键点击的概率决定,蓝色的鲜艳程度也与之相关,就像它填充的程度一样。

The amount of tilt was driven by the probability of the left click, and the vibrancy of the blue was attached to that alongside, like, how much it filled in.

Speaker 1

它会从光标的底部开始变化。

It would start kind of at the base of the cursor.

Speaker 1

随着倾斜加深,颜色会逐渐向顶端延伸,从淡蓝色变成非常明亮的蓝色。

And then as it was tilting in, it would just sort of, like, grow or stretch towards the tip and go from this, like, a nice light blue to a very bright blue.

Speaker 1

我们还可以利用显示屏的HDR功能,让这个效果稍微亮一些。

We can also use, like, the HDR parts of the display to, like, make that a little bit brighter.

Speaker 1

你肯定不希望每天要点击上千次的东西是这样的。

You don't want this you're gonna click a thousand times a day.

Speaker 1

所以,你确实希望这个效果能保持微妙。

So, like, you do want this to be subtle.

Speaker 1

我们不希望这个交互让人感觉沉重。

We didn't want this to feel like a weighty interaction.

Speaker 1

但我们的直觉是,让它保持连续性并直接映射解码器的输出,一方面能让交互感觉流畅自然,而不是生硬的离散操作。

But the intuition was that having it be continuous and map directly to the output of the decoder is something that one, let them just feel like a smooth fluid interaction as opposed to just this discrete thing.

Speaker 1

第二点,当功能失效时如何提供可见性始终是个挑战——既给用户也给我们自己提供可见性。

Two, this is always a challenge is when things aren't working, give us some visibility, both them and us some visibility into that.

Speaker 1

比如,你肯定不希望屏幕上随机出现点击却完全不知情。

Like, you don't want just random clicks happening on screen and you have no idea.

Speaker 1

第三点比较微妙:神经接口的有趣之处在于(其实所有接口都如此),你同时也在学习如何使用它。

And then the third thing is a more subtle point, but if the interesting thing about a neural interface, this is true about all interfaces, but you are also learning how to use it.

Speaker 1

当你在一个闭环系统中时,流程的最后一步是你实际观察输出,然后根据输出调整输入。

When you are actually in a closed loop way, again, the last step of that flow is that you actually observe the output and then change your inputs based on the output.

Speaker 1

正因为它是一个控制循环,看到从0.1概率到0.2再到0.3的每个阶段,能让你在潜意识里逐渐学会调整自己的行为,从而更好地点击。

So because it's a control loop, seeing each stage of that pipeline, when it goes from point one probability to point two to point three, gives you a way to subconsciously over time, learn to modulate your own behavior to click better.

Speaker 1

如果模型完全错误,这对你来说会是一件非常非常困难的事。

So if the model is just completely wrong, this is gonna be a very, very hard thing for you to do.

Speaker 1

但如果输出中存在细微的不准确,随着时间的推移就会产生某种共同适应的效果。

But if there are subtle inaccuracies in that output, there's kind of like a co adaptation that can happen over time.

Speaker 1

所以我们想确保这一点,比如在运动方面也是如此。

So we wanted to make sure that like, that's true of motion, for example.

Speaker 1

随着时间的推移,你可以学会更精确地移动它。

You can learn move it more precisely over time.

Speaker 1

这个学习过程很明显,想象一下如果你今天新长出一条手臂,你最初的动作肯定会非常怪异且不协调。

And that learning step, obviously, imagine like if you had a new arm, like today, like the very first things you do would be like super weird and uncoordinated.

Speaker 1

但随着时间的推移,你会学会精确控制这条手臂的每一个细微动作。

But over time you would learn to like really precisely modulate every single piece of that arm.

Speaker 1

但这确实是个挑战。

But that is a challenge.

Speaker 1

对吧?

Right?

Speaker 1

就像学骑自行车一样。

It's like learning how to ride a bike.

Speaker 1

我们希望将运动中的那种连续性同样应用到点击交互中。

We wanted the same sort of continuity that you have in motion to apply to the interaction of clicking.

Speaker 0

你们从参与者那里学到了哪些经验,改变了你们对点击行为的理解?

What were some of the lessons that you were learning from participants that evolved the way you thought about a click?

Speaker 0

比如,你们现在处于什么阶段?造成这种差异的原因是什么?

Like, where are you at now, and what is the reason for that delta?

Speaker 1

最大的变化其实是交互空间发生了很大改变。

The biggest thing was actually that the space of interactions changed a lot.

Speaker 1

所以我们从最初设想的一天只需一两次点击,扩展到了电脑操作中的所有丰富交互。

So we went from, let's say, just one click or two clicks that we wanted for the first day to the richer set of all of the things you do on a computer.

Speaker 1

滚动操作是个重要部分。

So scrolling is a huge thing.

Speaker 1

这与移动光标的方式截然不同,比如你实际滚动内容的方式。

That's very distinct from how you move a cursor, like how you actually scroll something.

Speaker 1

当然,拖拽操作是你认为应该直接生效的。

Of course, dragging and dragging is something you think should just work.

Speaker 1

拖拽是一种需要持续按住的动作。

A drag is something that you hold down over time.

Speaker 1

当我们按住时,就像存在某种阻力——实际上来自这些开关内部的弹簧——这种触感提示我们持续施压,而这在完全没有阻力的设备上显然是不存在的。

When we hold it, like there's some, again, some resistance that we actually feel right from the literal spring inside of these switches that is kind of our signal to keep applying pressure that isn't fully there with obviously something that offers like no resistance.

Speaker 1

从表征层面看,如果你观察神经数据,会发现有时它更像是先过渡到按下状态,然后再过渡到释放状态。

And it's also like representationally, if you actually look at like the neural data, what you see is like sometimes what it looks like is actually more of a transition into like clicking down and then a transition back into clicking up.

Speaker 1

对于拖拽光标来说,更好的模型可能是:你进入按下状态,再进入释放状态。

That may be a better model for a cursor that drags is it's almost like you get into the click down setting and you get into click up.

Speaker 1

反过来说,如果采用这种方式,所有非拖拽的点击操作都会变得迟缓很多。

The flip side is that if you go down that route, every other click that isn't a drag gets a lot, like, slower.

Speaker 1

这就像是因为你现在必须在这两种状态之间切换。

It's just like because you now have to, like, transition between these two states.

Speaker 1

这里可能有更好的建模方法,这绝对是很多人花大量时间思考的问题。

There may be a much better modeling approach here, and that's definitely something that a lot of people spend a lot of time thinking about.

Speaker 1

我们基本上希望你在第一天就能移动和点击。

We wanted basically on the first day for you to be able to move and click.

Speaker 1

由于你能做到这点,我们也希望至少再加入另一种点击方式。

And because you could do that, we also wanted to at least have one other click in there.

Speaker 1

坦白说,这只是个锦上添花的功能。

It's just a nice thing to have, frankly.

Speaker 1

右键点击对于快速调出菜单很有用,但并非必需。

Right click is like a pretty useful thing for just like popping over something, but not necessary.

Speaker 1

更多只是因为我们想测试一下。

It was more just because we wanted to test out.

Speaker 1

如果在混合操作中有多种点击方式,你能否可靠地在它们之间切换?

If you have multiple clicks in this mix, like, can you reliably switch between them?

Speaker 1

但随着时间的推移,你需要使用电脑进行的交互会变得更加丰富多样。

But over time, then there's a far richer space of interactions that you need to just use your computer.

Speaker 1

随着功能叠加得越多,那种简单的单一模型——你能看到物体是什么,光标内部在两者之间切换——就逐渐失效了。

And the more you stack in there, the less just having one simple model where it's just like you can see what the thing is and it's mixing between the two inside of the cursor, that just kind of fell apart.

Speaker 1

我们的第一位参与者在大约一个月后,其信号质量与第一天相比已经大不相同。

Our first participant, like one month or so into his journey, we had a very different signal quality than we did on the very first day.

Speaker 1

因此我们实际上退后一步,同时支持了悬停交互,因为我们发现在这种状态下,点击动作比移动更难解码。

And so we actually took a step back and also supported a dwell interaction because we found that the clicks in this regime were actually much harder to decode than the motion.

Speaker 1

这种能持续看到你在移动光标时每个细微动作的能力,让他移动光标比实际执行点击操作更容易。

Something about just that continuous ability to to see every slight movement you make in a moving cursor made it easier for him to move the cursor than it did for him to actually perform a click.

Speaker 1

这意味着我们需要这种渐进式控制,从单纯移动光标开始逐步扩展。

And so what that meant is we wanted this gradient of control that goes everywhere from just moving the cursor.

Speaker 1

如果你能移动光标,就可以用移动作为点击时机的信号。

And then if you can move the cursor, you can use motion as your signal for when to click.

Speaker 1

这就是悬停功能的作用。

That's what the dwell would do.

Speaker 1

如果你放慢速度并停下来,就可以基本保持在某个位置,然后通过这种方式执行点击操作。

If you slow it down and bring it to a stop, then you can basically hold that in a certain place and then perform a click that way.

Speaker 1

我们希望支持这种交互方式。

We wanted to support that.

Speaker 1

然后我们还希望支持叠加左键点击、右键点击、拖动、滚动、缩放等所有其他交互功能,随着用户逐步提升使用水平。

Then we also wanted to support layering in a left click, a right click, a drag, a scroll, a zoom, and all of the other interactions that you need to, like, you know, as you progressively ramp up.

Speaker 1

我们从仅用颜色显示两种点击的简单光标,发展到了这种更圆形的表现形式。

We went from just having this really simple cursor that could show you just using color, these two clicks, to the more circular representation.

Speaker 1

这就像一个圆环加一个圆点,几乎像个准星。

So this is like a circle and a dot, like a reticle almost.

Speaker 1

这种设计实际上让用户更容易看清自己点击概率的差异。

This actually made it a little bit easier to see the difference in your flick probability.

Speaker 1

所以在这个方案中,我们不再使用颜色,而是仅通过圆环的外半径来表现。

So in this world, we didn't use color anymore, but we used just this, the outer radius of the circle.

Speaker 1

当你带着点击意图瞄准某个目标时,它会逐渐缩小半径,聚焦到那个目标上。

As you were targeting something like you with the intent to click, it would sort of scale down, and focus in onto that thing.

Speaker 1

因此随着点击概率的提升,它基本上会收缩成一个点。

So it would collapse to a point, basically, the more you ramp up your click probability.

Speaker 1

或许因为我们的眼睛对运动比对颜色更敏感,这实际上更容易用来执行点击操作。

Perhaps because our eyes are more sensitive to motion than color, this was an easier thing, to actually use to perform a click.

Speaker 1

而且它也支持更自然地使用停留点击。

And it also supported just using dwell more naturally.

Speaker 1

圆形光标比现在的普通光标更容易透视观察。

The circular cursor was easier to actually see through than like a normal cursor is today.

Speaker 1

由于主要将其视为运动位置的信号,我们可以将其做得稍大些而不会遮挡实际要点击的内容。

And because primarily you're just looking at it as your signal of like where the motion is, we can make it a little bit bigger without it obscuring stuff you actually wanna click on.

Speaker 1

于是我们从传统指针样式转向了这种圆形样式。

So we went from this more traditional sort of pointer style to this sort of circular style.

Speaker 1

这也为更轻松地在不同交互模式间切换打开了大门。

And that opened also the door to going between different interaction interaction modes more easily.

Speaker 1

最终我们构建了不同的模式切换器——将光标猛甩到屏幕右侧就会进入轨道模式,这时可以用移动光标的相同模型选择新模式再退出。

So we ended up building a different mode switcher where you could sort of slam your cursor to the right side of the screen and then it would go on rails and you would use the same model that you used to move the cursor to pick a new mode and then come out.

Speaker 1

所以你会向右滑动,上下滚动,再向左滑动

So you would shoot right, scroll up or down, shoot left.

Speaker 1

这样做的好处是,一旦熟练掌握就能非常快速地操作,完全不同于传统的停留点击方式

What's nice about this is that it was something that could be really fast, basically, if you learn to use it well as opposed to something where you have to use your normal dwell.

Speaker 1

因为这样完全省去了选择模式的时间,整个过程一气呵成——就像通过神经肌肉记忆知道屏幕右侧的特定区域,然后快速向左返回

Because then there's no time spent actually selecting that mode, and it's just one fluid motion of, like, going to the right to the sort of part of the screen that you learn, like, through this sort of neural muscle memory of, like, where this mode should be and then popping out back left.

Speaker 1

当我们有了不同模式后,比如加入了拖拽和滚动功能,就能轻松改变准星的行为模式和视觉外观,让你清晰知道当前模式的交互逻辑

And then once we had different modes, we had, like, dragon there, we had scroll on there, we could easily change the behavior of, like, what this reticle is going to do and the visual appearance of, like, what it looks like to, like, make it obvious to you what it's going to do now that you're in this mode.

Speaker 1

我得说,很多这样的操作在使用传统光标样式时会更难实现,因为所有功能都集中在光标这一单一空间内。

A lot of that was, like, harder to do, I I'd say, with just, like, the conventional cursor style and having everything exist in one space, within the cursor.

Speaker 1

这里的权衡在于——随着我们模型的改进,这一点肯定会被重新审视——你必须切换模式才能改变交互方式。

The trade off there, and this is definitely as our models improve, something that I think we'll then go back and revisit, is, you do have to switch modes to actually switch the the way in which you do your interaction.

Speaker 1

所以,脑机接口的世界很有趣,如果模型完美的话,它总能如你所愿地运作。

So, yeah, the world of BCI is like an interesting one where there's like, if the model is perfect, it's always gonna do what you want.

Speaker 1

这就像是一句不言自明的话。

It's like an obvious statement.

Speaker 1

但在那样的世界里,你其实并不需要这些交互界面。

But in that world, you really don't need these interfaces.

Speaker 1

但我们的观点是,通往那个世界的路径更像是沿着阶梯逐步攀登,而非试图一次性完成所有事情。

But our bet is that the way in which you get to that world is, like, more of walking along this ladder than it is, like, trying to do everything at once upfront.

Speaker 1

我们本质上希望你现在就能用光标完成所有这些操作,即便实现方式更多是设计和工程优化,而非我们心目中完美的终极解决方案。

We want basically you to be able to use your cursor to do all this stuff today, even if the ways in which we get there are more of, like, design and engineering optimizations than they are what we think the perfect final solution is.

Speaker 0

你预见到那个阶梯上有哪些横档吗?

Are there rungs on that ladder that you anticipate?

Speaker 0

或者换个说法,是否存在某些特定的障碍或能力突破点,是你特别期待为这款产品带来的?

Said differently, maybe, like, specific barriers or unlocks in the capabilities that you're really excited to bring to this product?

Speaker 1

阶梯上最让我兴奋的横档——虽然这还是个开放性问题——就是:我们能否彻底取消光标?

The rung on the ladder that I'm most excited about, if this is something that we can do, I think it's, like, definitely an open question, would be, can we delete the cursor?

Speaker 1

如果你思考光标的本质,它其实就是你需要移动的意图焦点。

So if you think about what the cursor is ultimately, it is just like this focal point of your intent that you have to move around.

Speaker 1

90%的操作本质上只是'我想与屏幕上这个位置的元素交互'。

90% of the stuff you do is more just like, I wanna interact with this thing at this point on my screen.

Speaker 1

你真的需要一个光标来做这件事吗?

Do you need like a cursor for that?

Speaker 1

如果你能读取我想与屏幕上这个点交互的意图呢?

If you can read that intent that I have that I wanna interact with this point on my screen?

Speaker 1

可能不需要,我认为。

Probably not, I would say.

Speaker 1

光标对于直接操作任务来说非常出色。

The cursor is like amazing for direct manipulation tasks.

Speaker 1

所以当你真正想要直接操作某物时,光标会在操作时逐渐消失,这真的很棒。

So when you actually want to directly manipulate something, the cursor kind of fades away when you do that, which is really nice.

Speaker 1

你可以就像...感觉就像我在旋转这个东西。

And you can just like, it feels like I'm rotating this thing.

Speaker 1

感觉就像我在放大它。

It feels like I'm scaling it up.

Speaker 1

无论我对它施加什么变换,都感觉不到光标的存在了。

Whatever transformation I'm applying to it, it feels like the cursor isn't there anymore.

Speaker 1

这只是我的双手在与屏幕上的数字元素进行交互。

It's just my hands interacting with this digital element on screen.

Speaker 1

这是这个新世界中的关键交互模式。

That is a key interaction model in this new world.

Speaker 1

但就像第一步那样,大约90%的时间我只是想与某物交互,我认为不需要光标。

But like if that first step of just, like, the 90% of the time I'm just trying to interact with something, I don't think I need a cursor for that.

Speaker 1

所以,可以说这是这个阶梯的第一步。

So that's, like, one step of this ladder, I would say.

Speaker 1

这已经跑了不少次了。

That is quite a few runs out.

Speaker 1

在这个过程中,我个人对不需要切换模式的世界感到非常兴奋。

On the way to that, I'm personally very excited about a world in which you don't need to switch modes.

Speaker 1

就像我们最初设计的那样,你不仅可以看到点击概率,比如在这个光标内部,还能看到你想进行这种交互的方式。

Some form of that first design that we had where you can see not just the click probabilities, like, inside of this cursor, but the way in which you wanna do this interaction.

Speaker 1

举个例子,如果是类似这样的情况,我给你举个俗套的例子。

So for example, if it is something like, I'll give you like one cheesy example.

展开剩余字幕(还有 332 条)
Speaker 1

比如我想放大某个东西。

Like if I wanna zoom in on something.

Speaker 1

对吧?

Right?

Speaker 1

视觉上,我们仍希望为这种缩放操作提供某种形式的连续反馈。

Visually, we still wanna give you some form of continuous feedback about that zoom.

Speaker 1

光标几乎像减数分裂,像细胞分裂那样可以爆裂成两个手指点。

The cursor almost like meiosis, like cellular division could like explode into like two finger points.

Speaker 1

对吧?

Right?

Speaker 1

然后这两个点就能成为你进行缩放操作时的视觉锚点。

And then those could then be your visual anchor for how you're doing this zoom in interaction.

Speaker 1

这是阶梯的又一步。

That's another step of the ladder.

Speaker 1

如何突破当前交互模式的限制,实现无需切换模式的新型光标交互,这种模式切换正是我们模型能够学习的内容。

How do you go beyond basically the current interaction model into something that doesn't require switching modes to a new style of cursor, that mode switching is something that our model can actually learn.

Speaker 1

因此我们捕捉的是你的意图,不是那种笼统的‘我想与屏幕上这个东西互动’的高阶意图,而是具体到‘我现在想要拖动’的意图。

So we pick up on your intent, not in this case of just this crazy high order intent of I wanna interact with this thing on my screen, but rather at the specific I want to drag now.

Speaker 1

这会让人感觉像魔法一样,因为当你想要拖动时,它就会自动切换到拖动模式。

That could feel like magic because it's just like when you want to drag, it just switches to the drag thing.

Speaker 1

你不需要像现在这样通过模式切换器来传递意图——我们现在的方式是有一个模式切换器,还有一个快速切换功能,可以通过点击快速切换到当前应用的首选模式。

You don't have to channel that intent through how we currently do it, which is we have a mode switcher and we also have a quick switch where you can use one of your clicks to quickly switch to your preferred mode in this specific app.

Speaker 1

这样在功能上已经实现了90%的目标,因为大多数情况下你的第二个操作就是拖动。

So that gets us like, of course, functionally, like 90% of the way there because most of the time your second interaction is dragging.

Speaker 1

我很好奇你的操作习惯会是什么,但我的肯定也是这样,因为这可以按应用单独设置。

I'm curious like what yours would be, but that's definitely what mine would also be because you can do it on a per app basis.

Speaker 1

比如在Illustrator里,你经常需要缩放操作。

Like if in Illustrator, you're are, like, zooming.

Speaker 1

对吧?

Right?

Speaker 1

你完全可以重新映射这个功能——虽然这样能实现90%的目标,但我认为仍然缺少那种只有神经指向设备才能实现的独特魔力,这是物理指向设备做不到的,即我们能读取某种潜在的意图。

You can easily remap that in, like functionally, that gets you 90% of the way there, but it's still lacking, I think, the the actual magic that is unique to something like a neural pointing device that these physical pointing devices just can't do, which is that we can read some form of our underlying intent.

Speaker 1

我认为接下来的10个阶梯就是将真实的原始意图融入我们的体验中。

And I think the next like 10 rungs on this ladder are just bringing like the actual raw intent into our experience.

Speaker 1

所以到目前为止,目标只是让光标能够使用,并且达到与我们使用时相同的精准度。

So the goal so far, I mean, has been just to enable the use of the cursor and to enable it with the same fidelity that we can use it.

Speaker 1

我认为目前的记录,我们的第一位参与者设定了一个惊人的标准。

So I think the current record, our first participant set a wild bar.

Speaker 1

大约是每秒9.5次点击。

It was like nine and a half BPS.

Speaker 1

BPS只是我们实际可读取信息的一个度量指标。

BPS is just like a metric for the information that we can actually read.

Speaker 1

真的,你可以把它想象成一个分数,因为你可以用他们测量光标BPS的相同任务来测试。

Really, you can just think of it as just like a score because you can use the same task that they use to measure the BPS of like your cursor.

Speaker 1

比如,你可以用触控板或鼠标,按照他们的方法操作——点击屏幕上的位置,它基本上会测量你完成的速度和准确度。

Like, so you can use it with a trackpad, like a mouse, and just do what they do, which is, like, click on positions of the screen, and it measures basically how quickly you can do that and how accurately you can do that.

Speaker 1

是的,第一位参与者从信号质量下降的那个节点开始就设定了这个疯狂的标准,他不得不重新使用停留点击功能,因为他实在太爱使用他的神经光标了。

So, yeah, first participant set this wild bar from that first point where the signal quality had, like, dropped off and he had to go back to using dwell because he just loved using his neural cursor.

Speaker 1

他连续数月每天使用,操作水平突飞猛进,从最初大约4 BPS——最底层阶梯大概是2 BPS——一路提升。

He, like, used it every day for months and got insanely better at using it and went from, I think, something like four BPS, like, two at like, at the very bottom rung was, like, two BPS.

Speaker 1

他就这样一步步攀登,最终达到了9.5 BPS的水平。

And he just, like, walked his way up this ladder to, I think, nine and a half is where he ended up.

Speaker 1

作为参照,我的水平大概是10 BPS。

For context, I can do about, like, 10.

Speaker 1

这是我的个人记录。

Like, that's my personal record.

Speaker 1

说实话这很符合我的品牌调性,毕竟我用的是触控板。

I'm probably on brand, frankly, because I use a trackpad.

Speaker 1

但他的成绩最终稳定在9.5左右。

But he ended up around nine and a half.

Speaker 1

而今年我们的第十位参与者达到了10.5 BPS。

And then this year, our tenth participant did 10 and a half.

Speaker 1

最终数字我记得是10.38。

And it's, like, 10.38, I think, is the final number.

Speaker 1

主要目标是让我们能够以相同的精准度使用光标

The primary goal is to enable the use of the cursor at the same fidelity that we can use it.

Speaker 1

这对我们如此令人兴奋的原因是,光标不仅是您操作能力的焦点,更是您使用电脑的方式

The reason this is like so exciting for us is like the cursor beyond just being the focal point of your agency or whatever is also how you use a computer.

Speaker 1

所以如果您思考这意味着什么,这意味着我们用电脑来提高生产力

So if you think about what that entails, that entails like we use computers to be productive.

Speaker 1

我们用它们来表达自我

We use them to express ourselves.

Speaker 1

我们用它们来彼此交流

We use them to communicate with each other.

Speaker 1

我们还用它们来娱乐

And we use them to like have fun.

Speaker 1

这涵盖了人类体验中相当广泛的部分,而只需要一个真正流畅的光标就能实现

It's a pretty wide subset of the human experience that can be enabled with just a really sort of fluid cursor.

Speaker 1

所以这绝对是我们的目标

So that is definitely like the goal.

Speaker 1

这就是为什么我们要在这个充满不同权衡的空间中探索,只为实现某种形式的光标使用。

And that's kind of why we walk this space of like different trade offs just to enable some form of cursor use.

Speaker 1

因为我们觉得,显然还有更好的方案存在,这始终萦绕在我们心头——我们还能从这里走向何方?

Because we think that like, obviously there's this better thing out there that know, is in the back of our minds of, like, where can we go from here?

Speaker 1

但在这个案例中不去重新发明轮子的原因很简单:轮子是你驶上高速公路的工具,也就是现有系统。

But the reason, like, not to reinvent the wheel in this case is just because the wheel is how you get on the highway, which is this existing system.

Speaker 1

这绝对是我们的关注重点。

That's definitely the focus.

Speaker 1

但在此之上还有更多层级,比如DCI独有的特性——它们不仅仅是当前光标功能的简单投射。

But there are rungs, like, beyond that then, which is, like, there are things that are not just a projection onto a cursor that we use today that are very unique to DCI.

Speaker 0

我很想深入了解产品体验本身是如何随时间演变的。

I'm interested in learning more about just how the product experience itself has evolved over time.

Speaker 0

随着参与者使用频率的增加,他们的行为如何影响了实验设计?甚至如何改变了你们对产品定位的思考?

So as you're getting more and more usage from these participants, how are their behaviors shaping some of the experiments and just how you were even thinking about what the product needed to be?

Speaker 1

我们在系统中内置的一个通用后备方案是语音输入。

One fallback we built into the system in general was voice input.

Speaker 1

因此,仅通过简单语音指令就能重新校准模型或调整参数——虽然我们并未将其视为最终用户体验方案,但在早期系统极不稳定的阶段,这确实非常实用。

So just being able to say a simple command to recalibrate the model, to change some parameters of the model, not something we envision as our final sort of experience, but it was great in those early days when things were far less predictable.

Speaker 1

对他们而言,这是个极其可靠的备选方案。

It was a really reliable fallback option for them to have.

Speaker 1

事实上,我们的第三位参与者患有肌萎缩侧索硬化症(ALS)。

Our third participant actually had ALS.

Speaker 1

这意味着他们丧失了语言能力。

That means that they cannot speak.

Speaker 1

自然,语音输入在这种情况下就完全失效了。

And that means that, of course, voice input is not something that's going to work well here.

Speaker 1

这在某种程度上渗透到了我们交互界面的每个细节——那些我们曾视为理所当然的语音输入场景。

So this actually bled through to every edge of our interface in some sense, things that we had taken a voice input as for granted for.

Speaker 1

我们不得不更深入地思考:什么才是优质的备选方案?

We had to think a little bit harder about what a good fallback option would be.

Speaker 1

但最令人惊喜的是,我们的第三位参与者直接催生了一项功能,我们称之为'停车位'。

But I think the coolest thing that actually came out of this directly from our third participant was a feature that we call the parking spot.

Speaker 1

光标总是在某种程度上移动着。

A cursor is always moving to some extent.

Speaker 1

这是那种随着模型改进,希望永远不会再出现的情况之一。

This is one of those things in that bucket of like, as models improve, this will never be the case, hopefully.

Speaker 1

但短期内,你仍然需要使用它。

But in the short term, you still want to use it.

Speaker 1

所以会有一定程度的移动存在,模型可能错误猜测或将其与他们实际思考相关联,比如当他们与旁边的人交谈或在YouTube上看电影时。

So some amount of motion is going to be there that the model is just kind of incorrectly guessing or is correlated to them actually just thinking when for example, they're talking to somebody next to them or they're watching like a movie on YouTube.

Speaker 1

我们的模型会将某些活动错误标记为这里的移动。

There's just some amount of activity that our models are incorrectly labeling as motion here.

Speaker 1

你可以通过长期建模工作来解决这个问题。

You can solve this on the modeling side, longer term effort.

Speaker 1

短期而言,你只需要某种方式来停放光标。

Shorter term, it's like you just need some way to park the cursor.

Speaker 1

我们之前的参与者可以直接说'嘿'。

Our previous participants could just say, hey.

Speaker 1

关闭光标,它就会直接关闭,然后他们只需说打开就能再次启用。

Turn the cursor off, and it would just turn off, and then they could just say turn it on to bring it back on again.

Speaker 1

对于我们的第三位参与者,显然他们无法这样做。

For our third participant, obviously, they couldn't do that.

Speaker 1

因此我们需要想出某种方法,用光标来关闭光标,再用光标重新启用它。

So we needed to think of some sort of way to use the cursor to turn the cursor off and then use the cursor to bring it back on.

Speaker 1

我们尝试了几种方案,但最终采用的是这个称为'停车位'的方案,他们可以将光标'吃掉'或射到屏幕右下角。

We tried a few things here, but the final thing we landed on was this thing called the parking spot where they could just sort of eat their cursor or shoot their cursor into the bottom right of the screen.

Speaker 1

如果他们把它推过去,它就会停在那里。

And if they push it there, it'll park it.

Speaker 1

会弹出这个小东西。

It'll like this little thing will pop out.

Speaker 1

它会锁定光标,然后他们可以通过在该表面上做出手势来重新激活它。

It'll lock lock up the cursor, and then they can actually use a gesture within that surface to bring it back out.

Speaker 1

我们在里面应用了一系列小变换来保持它静止。

We apply a bunch of little transformations in there to hold it still.

Speaker 1

我们在其中模拟了重力效果。

We simulate gravity inside of it.

Speaker 1

他们仍然可以通过非常用力地推动或执行特定模式来操作。

They can still by pushing really, really hard or by actually going through a specific pattern.

Speaker 1

比如通过点点点的操作,他们可以完全自主地将光标重新拉出来。

So like a dot dot dot, they can pull the cursor back out fully on their own.

Speaker 1

这个设计非常酷,因为我们主要是为第三位参与者开发的。

So that was a really cool one, because we did that primarily for the third participant.

Speaker 1

但当我们推出时,前两位参与者当时都说:等等,这太神奇了。

But when we shipped that our first two at the time, we're like, wait, this is like amazing.

Speaker 1

实际上在最初几周,前两位参与者使用得更频繁。

And then they they actually were bigger, for the first few weeks.

Speaker 1

他们使用这个功能的频率甚至超过了第三位参与者。

Like, they were using it far more than our third party still was.

Speaker 1

从每位参与的个体身上,我们都能学到很多东西。

There's a lot you learn from, like, each individual participant that comes through.

Speaker 1

实际上大多数时候,他们反馈的内容往往也会渗透到其他所有人的体验中。

And most of the time, actually, what they give you feedback about often always bleeds out to everybody else's experience as well.

Speaker 0

像那样为光标模拟重力是什么意思?

What does it mean to simulate gravity on the cursor like that?

Speaker 1

你可以理解为,我们只是做了正确的数学计算来解决这个问题,但也可以想象当光标进入这个停车位时的情形。

You could think of it as like, we just do the right math to work this out, but you can think of it as like when the cursor goes inside this parking spot.

Speaker 1

几乎就像掉进了一个小山坡。

Almost like it falls into a hill.

Speaker 1

所以光标就像这样滚下来,然后停在那里。

So like the cursor kind of rolls down here, and then it's it's sitting there.

Speaker 1

我们显然可以调整那个山坡的深度。

We can obviously tune the depth of that hill.

Speaker 1

所以我们可以调整,比如是一个很陡的山谷,还是一个非常平缓狭窄的山坡。

So we can tune, like, if it's a really steep valley or if it's like just a really flat, narrow hill.

Speaker 1

但在实际操作中,这意味着他们需要更用力地推动才能让它滚上去,并在上升过程中保持动量。

But what it means in practice is that they have to push harder to roll it up there and, like, maintain momentum as they go up.

Speaker 1

模型在任何给定时刻的实际输出并非位置坐标。

The actual, like, output of the model at any given point is not a position.

Speaker 1

它实际上输出的是速度矢量。

It's actually, like, a velocity.

Speaker 1

准确来说,它输出的是一种微小的推力。

So it's actually, like, a nudge, so to speak.

Speaker 1

在这个情境下意味着,如果我们添加重力将光标向下拖拽,用户就需要手动将其滚出这个区域。

What that means in this context is, like, if we add gravity to drag the cursor down there, they'll have to actually roll it out kind of manually.

Speaker 1

但当我提到第三位参与者在最初两周使用频率较低时,是因为那个基于重力机制的版本对他完全无效——这引出了个非常有趣的技术难题。

But when I say that our third participant actually wasn't using it as much the first two weeks, it's because it that version of it, the gravity based approach, didn't actually work for him, which is a really interesting rabbit hole.

Speaker 1

和许多晚期渐冻症患者一样,他主要通过眼球追踪器与外界交流及操作电脑。

He like a lot of people with late stage ALS, he used an eye tracker as his primary way of, communicating with the world and using the computer.

Speaker 1

这意味着比如看电影时,他的视线会在屏幕上到处游移。

What that meant is that when he was, like, watching a movie, for example, your eyes are all, like, all over the screen.

Speaker 1

当时出现了个非常有趣的现象:他看电影时光标会突然弹出,甚至在他根本没注视屏幕时也会发生——即便设置了极强的重力约束。

It was a really interesting situation where when he was watching a movie, the cursor would just pop out when he's, like, not even looking at it no matter even with really, really strong gravity.

Speaker 1

但当他真正将视线聚焦在光标上,并试图在高重力环境下将其移动到停车位内时,推动它向上移动变得异常困难。

But when he really focused his eye on the cursor and tried to move it inside the parking spot with that high gravity, it was really hard to push it up.

Speaker 1

这确实非常有趣。

So really interesting.

Speaker 0

是啊。

Yeah.

Speaker 0

这种需要深入思考的细节层次确实很有意思。

That's that's a high level of depth that you have to think through that is really interesting.

Speaker 1

最终的解决方案其实是支持两种重力模式。

The solution there actually ended up being it supports, like, two modes where you can use it with gravity.

Speaker 1

实际上对于另外两位参与者效果非常好,以至于我们发现了新问题——比如有人反馈说‘我把光标停好正和人说话,结果屏幕就休眠了’。

And that actually for our other two participants, it worked so well that we found other issues where like one person was like, hey, I parked the cursor and I was talking to somebody and like my screen went to sleep.

Speaker 1

他进行了整整三十分钟的对话,期间光标完全没移动过。

So like he was having like a whole thirty minute conversation where the cursor doesn't move.

Speaker 1

在macOS系统里,如果光标不动,系统就会认为‘你没有在操作’。

And on macOS, if the cursor doesn't move, your system's like, oh, you're not doing anything.

Speaker 1

然后屏幕会变暗并关闭。

And it'll dim the display and then shut it off.

Speaker 1

所以在他们的情况下,效果太好了。

So in their case, it worked too well.

Speaker 1

在他这里,确实如此。

In his case, like yeah.

Speaker 1

至少在重力方面完全没有达到预期效果——当他看着光标时根本无法移动它。

It literally did not do with gravity at least what you'd expect at all, where when he looked at it, he couldn't move it.

Speaker 1

而当他移开视线时,却可能因为观看内容而产生巨大速度。

And when he looked away, he could have huge velocities as he's, like, watching something.

Speaker 1

所以我们最终为他采用了手势方案,效果非常好。

So we ended up going with a gesture for him, and that worked super well.

Speaker 1

这样他只需做出上下左右的简单动作。

So he could just do, like, an up, down, left, right.

Speaker 1

这些小圆点就会亮起来。

These little dots would light up.

Speaker 1

它会按顺序跟随这些点,然后他就可以这样把光标拖出来。

It would follow them in order, and then he would just, like, pull the cursor out that way.

Speaker 0

听你讲话很着迷,因为你痴迷于这些交互中最微小的细节。

It's fascinating to listen to you talk because you're obsessing over the smallest pieces of these interactions.

Speaker 0

我是说,从每一个可能的角度观察和思考光标,考虑所有能实现这个功能的不同方式。

I mean, observing and thinking about a cursor from every possible angle, all the different ways that we could make this possible.

Speaker 0

而最终目标基本上是要删除它。

And the end goal is to basically delete it.

Speaker 0

在某种程度上,删除你完成的所有工作。

To delete all of the work that you've done to an extent.

Speaker 0

你知道吗?

You know?

Speaker 0

这种矛盾张力真的很有趣,不是很多人有机会在这种状态下工作。

Like, that's a really interesting tension that not many people get to operate in.

Speaker 1

我认为最终目标就是打造一个绝佳的体验,并在每一步都做到这一点。

I think, ultimately, the goal is to just build an incredible experience and to do that in, like, every step of the way.

Speaker 1

所以,如果我们想要达到一个可以删除光标的世界,无论那意味着什么,我们的直觉是需要大量的数据才能实现。

So, like, if the way in which we get to a world where we could, let's say, like, delete the cursor, like, whatever that means, our hunch is that getting there is going to require, like, a lot of data.

Speaker 1

这需要对这些机制如何运作有更深入的理解。

It's gonna require a much more robust understanding of how these mechanics work.

Speaker 0

在那个世界里,光是想象就已经非常科幻了,衡量标准不再是所谓的普通人在电脑上能做什么。

In that world, it starts to get very sci fi to even think about, but the measuring stick is no longer what a, quote, unquote, normal person can do on a computer.

Speaker 0

你知道,你们已经突破了人与计算机交互可能性的天花板,达到了非常高的水平。

You know, you've blown the roof off of what what is possible in terms of interaction with computers at a a very high level.

Speaker 0

对吧?

Right?

Speaker 0

比如,在那个世界里,思考我在触控板上能多快完成某件事会显得很原始。

Like, thinking about how quickly I'm able to do something on a trackpad will feel archaic in that world.

Speaker 1

目前的交互模式与我们使用的非常相似,但没有理由必须如此。

Right now, the interaction model is, again, very similar to the ones that we use, But there's no, like, reason it has to be.

Speaker 1

因此我认为这个世界会出现新的交互模式,它们与我们用物理手部动作来表达的方式截然不同。

So I think there will be new interaction models in this world that are, yeah, just very different than having to use our, like, physical hands to, like, articulate something.

Speaker 1

我不确定那是否算过时,但我觉得肯定会截然不同。

I don't know if it would be, like, archaic, but I think it would just be very different.

Speaker 1

希望它会变得更加自然,更加直接。

Like, it'll it'll just be hopefully a lot more natural and hopefully a lot less indirect.

Speaker 0

作为比普通听众更常思考这个特定领域发展方向的人,你有什么特别关注的具体方面吗?

Anything specific that you find yourself thinking about as someone that probably spends more time pondering where this specific world is heading than the typical person listening, for instance?

Speaker 1

对我来说有几个方面。

I think there are a few for me.

Speaker 1

需要说明的是,这些只是我个人感兴趣的内容,并不一定是我们正在研究的项目。

The caveat being that, like, this is definitely just, like, the stuff that I'm excited about and not necessarily stuff we're working on.

Speaker 1

而且我觉得有趣的是,在NERLY公司,你会根据谈话对象得到百万种不同的答案。

And I also think, like, the fun thing is, like, at NERLY, you'll get a million different answers based on who you talk to.

Speaker 1

但对我而言,很多想法都源于十五年前那项最初激发我对这个领域兴趣的研究,我认为最丰富的元素是视觉图像。

But I think for me, it really a lot of it goes back to that study from fifteen years ago that first sparked my interest in this field, which is, like, I think the richest things in my mind are visual imagery.

Speaker 1

同时我认为,无论是作为软件设计师还是年轻时想成为电影制作人,计算机的大部分工作、魔力以及繁琐之处,都在于将脑海中的图像——无论是通过软件、After Effects还是Final Cut——花费很长时间具象化为这种人们更容易理解的视觉产物。

I also think that, like, both, like, as a software designer and as, like, a wannabe filmmaker when I was younger, like, that's where so much of the work so much of the magic of a computer is and also so much of the tedium of a computer is, is in the this process of taking an image in your mind and spending a very long time, whether it's in software, whether it's in After Effects or Final Cut, to articulate that into this visual thing that it feels like people just understand more easily.

Speaker 1

一张图片胜过千言万语,基本上是这样。

A picture is worth a thousand words, basically.

Speaker 1

这方面的事情正是我最期待的,你知道,展望非常遥远的未来。

That side of things is I think what I'm most excited about, you know, looking out into the very distant future.

Speaker 1

这意味着什么,我们经常讨论如何弥合人与人之间的共情鸿沟,但这感觉像是一个更具体的版本——我可以向你展示我的感受,而不只是告诉你。

What that would mean, we talk a lot about, you know, bridging this empathy gap between folks, but, like, that feels like a much more tangible version of that where I can show you how I'm feeling instead of telling you how I'm feeling.

Speaker 1

同时也适用于世界上人们想要创造的所有东西。

And then also just for all of the stuff that people wanna make in the world.

Speaker 1

我认为每个人都拥有并共享着这个丰富的内心世界。

There's this rich interior space that I think everybody has and everybody shares.

Speaker 1

有些人告诉我'我不太有创造力'时,我总觉得很有趣。

I always think it's, like, funny when some people tell me, like, oh, like, I'm just like not very creative.

Speaker 1

你晚上会做梦吗?

Do you dream at night?

Speaker 1

如果他们回答是,我就会说:看,你的大脑能够创造和生成多么丰富的内容。

And they're like, if so, that, like, look what your mind is capable of making and generating.

Speaker 1

我认为这在未来将是一项极具赋能性的技术,届时你基本上可以坐在电脑前

I think that is a hugely enabling thing in the future where you can sit down on a computer basically.

Speaker 1

在这个世界里,电脑上的模型能代表你的意图行动,比如说,在未来的某个时间帧里

And in this world where a model on that computer can act on behalf of your intent in, let's say, you know, far down the line, a frame.

Speaker 1

所以我们现在称之为'氛围编程'的技术,在未来,模型生成你描述的作品所需时间,我认为将缩短至一个时间帧

So what we today call vibe coding, you know, in the future, like the time that it takes for a model to actually produce an artifact that you described, will I think come down to a frame.

Speaker 1

而另一方面,现在的瓶颈是什么呢?

And the flip side of that is then what is the bottleneck here?

Speaker 1

关键在于你能否准确表达自己想要什么

It's you actually articulate what it is that you want.

Speaker 1

在艺术和工作等所有领域,这将是一种非常激动人心且奇妙的体验——你可以坐在电脑前,基本上就是和它一起做白日梦

In all domains in the arts, and in work, this will be a very exciting and wild experience where you can just sit down on a computer and basically daydream with it.

Speaker 1

屏幕上会即时呈现出你脑海中想法的直接延伸

And things will pop up on screen that are a direct sort of extension of what you have in your head.

Speaker 1

如果这一切都能在一个时间帧内完成,那会是什么样子呢?

And what would that look like if that came down to just being one frame?

Speaker 1

我真的不知道。

I truly don't know.

Speaker 1

但我认为那将是一个非常、非常令人兴奋的世界。

But I think that that's a really, really, exciting world to be in.

Speaker 1

我们正在研究一种叫做'盲视'的技术。

We are working on something called blindsight.

Speaker 1

这是这个阶梯的最初一级,即为那些失去视力的人找到某种方式,让他们重新看见世界。

That is the earliest rung in this ladder, which is that for people who no longer have vision, some way for them to see again in the world.

Speaker 1

这意味着我们的植入体将位于大脑的视觉区域,而非当前植入体所在的运动控制区。

What this would mean is our implant sits in the part of the brain, not for motor movement, which is where our current implant is, but for vision.

Speaker 1

我们实际上可以从他们佩戴的眼镜中读取所见内容,然后在大脑相应区域创建正确的刺激模式,试图在他们的意识中重现某种图像形态。

And we can actually read from like, let's say a pair of glasses that they wear, what they're seeing, and then create the right stimulation pattern in that part of the brain to try to recreate some form of that image in their mind.

Speaker 1

这显然令人难以置信。

Obviously, is incredible.

Speaker 1

但同时这也像是一个真正狂野的设计空间,因为初代版本——用个粗糙的比喻来说——会像是雅达利游戏机,而非PlayStation 5那样的高级设备。

But also it is like a truly wild design space because the first versions of this are gonna be, to use a really crude metaphor, they're gonna be like Atari as opposed to being like, you know, a PlayStation five.

Speaker 1

因此,相对于视觉信息实际上的丰富程度,我们能够在大脑这个区域通过电极数量创造的视觉保真度将会非常有限。

So the visual fidelity that we can actually create in terms of just the number of electrodes that are in this part of the brain is going to be very small relative to how rich visual information actually is.

Speaker 1

这就形成了一个完全独立的设计领域:如何在这个领域内‘真实’地重建一个图像?

So then that's its own completely distinct design space of like, how do you recreate an image that is true to life, quote unquote, in this domain?

Speaker 1

哇。

Wow.

Speaker 1

这个图像的哪些特征是需要在这里突出表现的关键要素?

What features of that image are like the right ones to highlight here?

Speaker 1

本质上,你想给人们提供哪些可调节的参数?

What knobs do you want to give people basically?

Speaker 1

因为他们应该对如何感知世界拥有某种形式的控制权。

Because they should have some form of control over how they see the world.

Speaker 1

这可能会成为某种既科幻又酷炫的功能。

That can be something that's just like sci fi and cool.

Speaker 1

比如能够对物体进行变焦观察,对吧?

It's like being able to zoom into things, right?

Speaker 1

但它也可能是更为微妙的东西,比如——借用更多比喻来说——抖动处理。

But it could also be something far more nuanced in terms of how, you know, you want to for example, just to borrow more metaphors, dithering.

Speaker 1

如果你熟悉抖动处理,这是一种在八十年代像素数量有限的条件下,通过利用人类感知特性设计的算法,在低分辨率空间中尝试重现我们感知到的特征(如阴影和纹理)的方式。

If you're familiar with dithering, it's like one way of in the eighties, when we were in this regime of, like, low pixel counts, to try and recreate features that we perceive through, you know, algorithms that apply that basically are artifacts of our perception and use this, like, lower sort of resolution space to still try and create some sense of, for example, like, shadows and some sense of, like, texture in this world.

Speaker 1

那么,在这个领域中抖动处理会是什么样子呢?

So, yeah, what would dithering look like in this domain?

Speaker 0

我在节目中听过很多内容,但这绝对是我听过最有趣的设计可能性空间之一。

I've heard a lot on this show, but that's one of the most interesting design opportunity spaces that I've ever thought of.

Speaker 0

就像这个概念在我脑海中根本不存在,直到你分享了它。

Like, it just didn't exist in my brain until you shared that.

Speaker 0

这真是太神奇且引人入胜,不仅因为它的新颖性,更因为你正在产生的影响确实令人难以置信。

And it's amazing and compelling, not just because of its novelty, but also just the impact that you are having is really incredible.

Speaker 0

我知道我们在开始录制前稍微提到过,但我正在阅读一些参与者的故事。

And I I know I mentioned this just a little bit before we started recording, but I was reading some of the participant stories.

Speaker 0

我看到诺兰——首批参与者之一——的故事,他正在重返校园,还在互联网上找到了一份工作。

And I was reading how Nolan, one of the first participants, got you know, he's going back to school, and he got a a job on the Internet.

Speaker 0

这真是太棒了。

And it's just amazing.

Speaker 0

真的,这让我深受感动。

Like, it moved me deeply.

Speaker 0

所以你们所做的一切简直是最鼓舞人心的。

And so what you all are doing is just about as inspiring as it gets.

Speaker 1

我完全同意。

I fully agree.

Speaker 1

我认为这正是我们团队乃至整个公司的人每天起床工作的动力所在。

I think this is the stuff that people get out of bed for on our team and definitely throughout the company.

Speaker 1

我们非常幸运能身处这样一个世界,现在有超过10个人真正在使用这个产品。

We are so privileged to be in this world now where we have 10 plus folks who are actually using the thing.

Speaker 1

我们称他们为'神经结',就像宇航员一样。

We call them, like, the neural knots, because, like, astronauts.

Speaker 1

这是我们共同探索的全新领域。

It's a completely new space that we're exploring together.

Speaker 1

他们显然已经做到了,比如回到光标过去的样子,以及现在功能运作的方式。

They've obviously done, like, back to that point of, like, what the cursor used to look like and how things work now.

Speaker 1

当他们能真正告诉我们使用感受,以及哪些部分确实需要优化、哪些部分无关紧要时,情况就发生了巨大变化。

Things have changed dramatically once they could actually tell us how things feel and what pieces of that, like, we actually do need to optimize, what pieces of that don't really matter.

Speaker 1

但归根结底,这能对他们生活产生的影响,对我们所有人来说都极具启发性。

But, ultimately, like, yeah, just the impact it can make on their lives has been super inspiring for all of us.

Speaker 0

假设有人在听,并且被这段旅程所激励。

Let's say that somebody is listening and they're inspired by this journey.

Speaker 0

他们想参与其中。

They wanna be a part of it.

Speaker 0

你们已经开放了第二个设计工程师类型的职位。

Y'all have opened up a second, you know, design engineer type role.

Speaker 0

能详细说说这个职位具体是做什么的吗?

So could you share a little bit more about, like, what's that role going to be like?

Speaker 0

我相信有听众会想:这听起来很棒,但我还不太清楚具体工作内容。

I'm sure someone is listening and they're like, yeah, that could be really cool, but also I don't a 100% get what that would look like.

Speaker 1

我的意思是,这个职位基本上涵盖了我今天谈到的所有内容,只是工作量会更大。

I mean, the role would be a lot of what I think I've talked about today, just a lot more of that.

Speaker 1

我们现在处于这样一个阶段:已经有超过10个人每天都在积极使用这个产品。

We're at this phase now where we have, yeah, 10 plus people that actively use the thing every day.

Speaker 1

他们每周累计使用时长达到数百小时。

They use it for, like, hundreds of hours a week collectively.

Speaker 1

其中很重要的一部分工作,当然就是持续为他们优化使用体验。

And one piece of that is, of course, just, like, actually just continuing to make that a great experience for them.

Speaker 1

由于我们团队规模很小,我们会高度聚焦于体验中最关键的部分——那些我们投入最多精力去完善的功能。

We generally because we are such a small team, we are super focused about what pieces of that experience, like we spend the most time on making great.

Speaker 1

我们对'优秀'的标准定得非常高,而且希望在整个产品体验中始终保持这个标准。

And we have very high bar for like what great is, and we wanna maintain that across that whole experience.

Speaker 1

所以首要任务就是让这个日常使用产品变得惊艳,并持续保持这种惊艳感。

So one piece of it is literally just this thing that people use literally every day, making that amazing and continuing to make that amazing.

Speaker 1

另一部分工作则是探索我们尚未触及的所有可能性领域。

A second piece of it is all of the rungs in the ladder that we have not explored yet.

Speaker 1

现在我们有了可以实际尝试并给出反馈的团队成员,这个过程实际上需要你从头到尾完整走一遍,这意味着你可能对事物运作方式有个初步想法或直觉,也就是所谓的假设。

The process of doing that now that we have folks who can actually try stuff and tell us is really just like, it's something you have to kind of go end to end on, which means that there may be some initial idea or hunch you have about how things might work, what you might call a hypothesis.

Speaker 1

从假设到真正做出可供人们使用的产品,中间还有大量实际工作要做。

There's a lot of actual work in between that and making it something that somebody can actually use.

Speaker 1

显然这其中涉及大量的设计工作。

There's obviously a lot of like design work there.

Speaker 1

同时还需要通过工程手段将其转化为用户首次可以使用和体验的产品。

And there's also just like the engineering work of turning it into something that they can use and try out for the first time.

Speaker 1

微妙之处在于,如果他们需要主动使用和尝试,那么首次体验未必能提供最有价值的信息。

The nuance is also that that first experience they have with it is not necessarily going to be the most informative if it's something they actually actively have to use and try.

Speaker 1

我们参与者的优秀之处在于,他们非常愿意直接拿东西去试用和体验。

What's awesome about our participants is like they're so down to just like take something and run with it and try it.

Speaker 1

他们使用一天和一个月的反馈质量会有显著差异。

And they will give you such great feedback after using something for like a day versus like a month.

Speaker 1

有时候要获得有意义的信号来判断方向是否可行,需要的不仅仅是一个原型。

Sometimes it's also not just a prototype to actually get a meaningful signal on, is this a direction we want to go in it?

Speaker 1

比如,这个功能能行得通吗?

Like, is this a feature that will work?

Speaker 1

你需要更进一步,不能仅仅满足于在单次用户测试中勉强可用的方案,而要把它变成用户能持续使用一个月的东西。

You kind of need to go beyond, yeah, just something that will work in a hacky way in one session with a participant, and it turn it into something that they can live with for a month.

Speaker 1

现实情况是,这些方案中的绝大多数最终都不会成功。

The vast majority of those things also, like, will not pan out is the reality of it.

Speaker 1

所以目前我们处于这样一个阶段:既要完善一组核心功能,因为我们希望第一版产品能问世;

So right now, we're in this phase where there's both a core set of features that we know that we wanna make great because we want that v one to exist in the world.

Speaker 1

同时还要探索更多可能性,这需要具备从设计到工程全链路推进的能力。

And also a lot of blue sky around what this could be that requires an ability both to just, like, kind of go end to end in both design and engineering.

Speaker 1

这就是这个角色的职责。

It's the role.

Speaker 1

但也要对某些不确定性保持适应。

But also be somewhat comfortable with things.

Speaker 1

实验性的本质意味着存在大量不确定性——比如这个东西最终能发布吗?

It's the fact that it's experimental, meaning that, like, there will be a lot of uncertainty into, like, will this thing actually ship?

Speaker 1

因为你希望有人能实际使用它,所以还需要投入大量工作来真正以某种形式发布,让他们可以试用。

And because you want somebody live with it, there's also a lot of work that needs to go into actually shipping it in some form so they can try.

Speaker 0

这个蓝海部分让我觉得非常有趣,因为光是听你讲了这么一会儿,意思就已经非常明确了。

The blue sky piece is really interesting to me because even just listening to you talk now for a while, I mean, it's so clear.

Speaker 0

你们正在做的事情与浏览Mobbin并试图拼凑现成组件、寻找世界上已有解决方案的做法完全相反。

What you're doing is the complete opposite of looking on Mobbin and trying to piece together the right pieces, the right things that already exist in the world to, like, figure out a solution.

Speaker 0

你知道吗?

You know?

Speaker 0

你们是真正从第一性原理出发开展工作,名副其实。

Like, you are really working from first principles in the truest sense.

Speaker 0

那么我的问题是:你会关注候选人的哪些信号,才能让你达到确信的程度,觉得'没错'?

My question then is, what are the signals that you would even look for in a candidate that would get you to a confidence level where you're like, yeah.

Speaker 0

是的。

Yeah.

Speaker 0

他们确实具备推动进展、帮助我们找到下一阶段发展阶梯所需的能力?

They actually have what it takes to move the needle and help us find some of those next rungs on the ladder?

Speaker 1

我们最优先优化的是使用界面时的实际感受。

The thing we optimize most for is feeling when it comes to what it's actually like to use the interface.

Speaker 1

因此,尽管我们试图对无法亲身体验的空间构建模块保持原则性,但总体目标始终是创造真正令人惊叹且鼓舞人心的作品。

So as much as we try to be principled about what the building blocks should be for space that we cannot experience ourselves, I think it's just a general ambition to just make something that's like really awesome and inspiring.

Speaker 1

我想我们都——是的,显然间接受到了参与者的启发,我们希望他们使用这个神奇新玩具时的体验是惊艳的。

I think we're all, yeah, obviously like indirectly so inspired by our participants, and we do want what it feels like for them to use these this magical new toy to be amazing.

Speaker 1

任何对这些事有明确渴望的人,在实践中往往意味着:即便是明知可能会删除的内容,我们也会投入大量额外工作去打磨它。

Anybody who has a clear desire to do those things, that's usually what that means in practice is even for something that you know that you're probably going to delete, there's like a lot of extra work that goes into just making that amazing.

Speaker 1

这就像许多公司在产品即将正式发布前的打磨与调优阶段所做的工作,我们会尽可能将这些工作融入实验环节,因为用户终将使用它。

The equivalent of kind of like the polishing phase and the tuning phase that you'll have at a lot of other companies when something is actually, you know, going to ship in a calendar year, we try to put as much of that as we can into just an experiment because a person will use it.

Speaker 1

此外,原型开发能力在这个环节中至关重要。

Also, this is where the prototyping ability, like the engineering side of the role is so key.

Speaker 1

这里的技术栈非常广泛。

The stack here is very wide.

Speaker 1

我认为最紧密的协作实际上发生在机器学习工作与实际界面开发工作之间。

I think the tightest interplay is really between the machine learning side of the work and the actual interface side of the work.

Speaker 1

因为机器学习这一侧,归根结底,至少目前来说,我们需要找到一种方式来标注某些内容。

Because the machine learning side, ultimately, some form of that, at least right now is, well, we need to figure out a way to label something.

Speaker 1

所以某种程度上,标签最终与他们在屏幕上看到的内容紧密相关。

So like the label is ultimately what, on some level is very tied to what they see on screen.

Speaker 1

这个标签能在多大程度上捕捉到真实意图,比如他们当时的真实想法。

How well that label captures the real intent here, like what they were really thinking at that time.

Speaker 1

模型的能力决定了界面可以实现的功能。

What the models can do defines what the interface can do.

Speaker 1

界面设计在某种程度上极大地限制了模型的能力范围。

And what the interface does in some ways heavily biases what the models can do.

Speaker 1

因此这两者之间存在着非常直接的关联,这使得当前的主要瓶颈变成了:能否构建出一个端到端的测试方案?

So there's kind of like a very direct relationship between those two things especially that makes the primary bottleneck here just like, can you build out something to try these things end to end?

Speaker 1

因为很多工作仅停留在离线数据层面——比如收集数据后仅进行离线分析——对于UI相关的工作来说基本是条死胡同。

Because a lot of stuff is just the offline data, meaning like trying to collect data and then just run your analysis offline, generally is a dead end for anything on like the UI side.

Speaker 1

这意味着你必须构建一个真正完全交互式的系统才能有效验证,并可能需要与建模团队协同工作。

So it means you have to make something that is like actually fully interactive to actually validate this well, and potentially also work with folks on the modeling side.

Speaker 1

如果你有一个非常大胆的新想法想要尝试,那会涉及很多方面。

If you have a really wild new idea you want to try, that involves a lot.

Speaker 1

这包括收集数据的任务,也包括最终的控制界面。

It involves a task to collect the data, it involves some final interface so that they can control it.

Speaker 1

再次强调,理想情况下这需要他们能够独立使用,并提供更丰富的信号来评估其效果。

And again, it involves ideally something that they can live with independently and give you a much richer signal on how well it's going to work.

Speaker 1

因此,我们需要那些对所有这些充满热情的人,他们还要热衷于让每个实验都尽可能出色,这样我们才能确信,当我们遇到瓶颈时,不是因为实验本身只是个最小可行产品。

So people that, yeah, are excited about doing all of that and also are excited about making each individual experiment you run as amazing as it can be, such that we have confidence that when we hit a dead end, it's not because it was, for example, like an MVP.

Speaker 1

我通常认为,MVP(最小可行产品)的心态有时会适得其反,因为你基本上只做了最低限度的工作,得到的结果也很平庸。

I generally think that like the mentality of like an MVP can sometimes be quite self defeating, because you've basically done the absolute minimum you possibly could have done and gotten like a pretty mid signal on something.

Speaker 1

我们尽力尝试的每一次尝试,都会全力以赴,因为最终的结果很大程度上决定了我们未来的方向。

The shots that we try to shoot, we try to, like, shoot them as best we can because the the results that come out of that very much dictate ultimately the direction we're gonna go in.

Speaker 0

能否从你的亲身经历中举一个例子,说明某个直觉如何引导出实验过程,帮助人们更好地了解在Neuralink工作的日常是怎样的?

Is there an example of a hunch that led to some experimentation process that you could talk through from your own experience that would help people get a little bit better sense of what it is like day in a life of someone working on Neuralink?

Speaker 0

如果还能举例说明在整个实验中如何注重细节,那就更好了。

And maybe if there are also examples of what it looks like to sweat the details throughout that experiment, that would be great too.

Speaker 1

这里有个很好的例子,就是我们目前称之为身体映射任务的项目。

One good example here is what we currently call the body mapping task.

Speaker 1

当前这个任务的结构是:我们渲染了一个真实的三维手臂,并且有完整的渲染管线来绘制这个三维手臂。

The structure of this task right now is there's an actual, like, three d arm that we render, and we have a whole rendering pipeline to draw this three d arm.

Speaker 1

这种方法意味着我们可以将你的注意力引导到非常具体的部位,然后结合我们提供的实际指导——包括指导中给出的视觉指示器。

The approach there means that we can draw your focus to very specific pieces of it and then combine it with the actual guidance that we tell you with, like, visual indicators in the guidance that we give you as well.

Speaker 1

那么这个任务具体是做什么的呢?

So what does this task do?

Speaker 1

这是我们最初规划时的首要事项。

This is the very first thing when we sort of spec this out.

Speaker 1

这将是参与者做的第一件事。

This would be the very first thing that participants does.

Speaker 1

在他们实际进入体验阶段,首次校准植入物并实现光标移动之前,他们需要先探索这个广阔的空间——你现在要重新想象移动你的手臂,这个你目前无法移动或仅有非常有限残余运动能力的手臂,并感受什么方式对他们有效。

And before they actually go into the part of the experience where they sort of calibrate their implant for the very first time and get to a moving cursor, they just explore this like wide space of like, you are now going to imagine moving your arm again, this arm that like you currently cannot move or have very limited residual motion for and get a feel for just like, what works for them.

Speaker 1

比如,哪些动作对他们来说仍然感觉直观,即使他们无法很好地移动它,哪些则不然。

Like, what kind of motion still feels intuitive to them even though they can't move it well, which ones don't.

Speaker 1

同时也能让我们获得——不是感受,而是信号——关于哪些动作我们能够准确读取的信息。

And then also get a feel get us not a feel, but a signal for what motions can we actually read really well.

Speaker 1

所以当他们想象做这些动作时,那阵神经活动会是什么样子?

So when they imagine doing this, like, what does the burst of activity look like?

Speaker 1

哪些动作在这里表现更好,哪些更弱?

And which motions seem to be better here and weaker?

Speaker 1

然后在两者之间进行比较,比如从这一系列动作中选择一个——你知道的,上下左右移动手臂,上下左右转动手腕,还有大概四种其他可以用来移动光标的动作——从而了解对他们来说感觉良好的动作与我们实际能解码的动作之间有多少重合。

And sort of compare between the two, like, pick one from this wide set of, like, you know, moving your arm up, down, left, right, moving your wrist up, down, left, right, and, like, four other things that you could presumably use to move the cursor, getting some sense of the overlap between what feels good for them and also what we can actually decode.

Speaker 1

我们最初的直觉主要是,在屏幕上看到那只手臂会让初次体验比我们单纯告诉他们'嘿,试试这样做'自然得多。

The hunch here was primarily that, like, seeing that arm on screen would make that first experience far more natural than us just telling them, hey.

Speaker 1

比如,试着这样做。

Like, try doing this.

Speaker 1

这会给他们一个可以观察和对照的参照物。

It would give them something to basically, like, look at and, like, cross reference.

Speaker 1

而这里的细节基本上就是,好吧。

And then the details here were basically like, well, okay.

Speaker 1

那么,我们想怎么做这件事呢?

Well, how do we wanna do this?

Speaker 1

第一,我们可以尝试预先渲染所有这些动作。

One, we could, like, try and pre render all of these motions.

Speaker 1

这样可能行得通。

Like, that will probably work.

Speaker 1

但如果我们能直接解码那只手臂的动作呢?

But what if we could actually just decode that arm then?

Speaker 1

为什么要先处理这只手臂,然后再尝试控制光标呢?

Like, why take this arm and then try to, like, go down to the cursor?

Speaker 1

如果他们已经回到只需观察那只手臂的世界,从某些方面来说,这比想着通过移动手臂来控制光标要自然得多。

If they're already back in this world of just looking at that arm, in some ways, that's a far more natural thing to do than trying to think about moving your arm to then move a cursor.

Speaker 1

所以这里真正的直觉是:如果我们能做到这点,他们是否也能像操纵木偶一样控制这只手臂?

So the real hunch here was if we can do this, can we also like, can they also puppet this arm, basically?

Speaker 1

要实现这点,需要极其庞大的工程工作,因为我们不能再使用预先渲染的动画了。

To do that, that was an extraordinary amount of engineering work because we can't just use a pre rendered animation anymore.

Speaker 1

我们需要实际渲染一个3D手臂。

We need to actually render like a three d arm.

Speaker 1

我们需要将其网格化。

We need to mesh that out.

Speaker 1

我们需要建立一些着色管线,这样才不至于让效果看起来很滑稽。

We need to have some shading pipeline so that we can like make that not look goofy.

Speaker 1

显然,屏幕上出现3D手臂会产生巨大的恐怖谷效应。

There's obviously a huge uncanny valley in a three d arm on screen.

Speaker 1

所以我们希望它基本上看起来具有最少的细节,让你能感受到手势的实际含义,而不需要那些关于手臂是什么的冗余信息。

So we wanted that to just look basically like have the minimum amount of detail for you to, get a feel for what the gesture actually is without like all of the redundant information about like what an arm is.

Speaker 1

同时我们也希望它感觉自然。

And we wanted to also like feel natural.

Speaker 1

所以我们基本上不想使用骨架结构。

So we didn't want to use like a skeleton basically.

Speaker 1

你也可以设计成能看到所有骨骼的样子,但我们觉得那样做出来的体验不会很酷。

Like you could also do something where you see all the bones, but like that didn't feel like it was going to be a really cool experience to us.

Speaker 1

最后,我们额外添加了一点完全没必要但觉得超酷的魔法效果——现在你有了这只手臂,我们还需要配上文字说明动作类型。

And then finally, like one extra bit of magic that we absolutely did not need to do, but we thought it'd be really cool if now you have this arm and we also need to give you some text to like tell you what the kind of motion is.

Speaker 1

比如要做个‘捏’的动作时,我们就需要弹出提示说‘嘿,捏这只手臂’。

So for example, like if you're gonna do like a squeeze, like we need to So, like, pop up something that says, hey.

Speaker 1

捏这只手臂。

Squeeze this arm.

Speaker 1

这效果会很棒,因为文字就在屏幕上,你的视线本来就会聚焦在那里。

It'd be really cool if because that's on screen, so your eye is going to already be looking there.

Speaker 1

如果文字本身能配合手臂动作动态呈现,就能强化整体效果。

If the sort of text itself can embody that motion alongside the arm just to, like, reinforce that whole thing.

Speaker 1

于是我们做了这些狂野的文字动画——当你做捏的动作时,‘捏’字会聚焦放大,背景所有内容都会后移。

Then we did these wild text animations where, like, when you do the squeeze, like, the squeeze itself kind of, like, focuses in so everything in the background sort of, recedes.

Speaker 1

然后这个‘捏’字还会做动态字体效果,

And then the squeeze actually, like, we do kinetic type.

Speaker 1

让它完全随着捏的动作同步收缩。

So it, fully squeezes in alongside the motion itself.

Speaker 1

这就是一个我们完全没必要做的例子。

So that's an example of something we totally did not need to do.

Speaker 1

但它就是很酷。

It's just cool.

Speaker 1

这里的直觉其实是:如果这个方案有效,基本上,如果我们能通过屏幕上显示这只手臂并使其视角与用户位置相匹配来建立这种身心连接,那么或许我们就能创造一个简单的想象空间,让他们在脑海中想着‘好,现在要挤压手臂’。

The hunch here was actually that if this works, basically, if there is this mind body connection that we can induce by then just having this arm on screen, having the perspective of it feel right relative to like where they are, that maybe we can actually have this simple imaginary space where in their head they're just like, okay, now squeezing the arm.

Speaker 1

我完全不用考虑这些关节问题。

I'm not thinking about all these joints.

Speaker 1

我并不是在考虑移动每根手指

I'm not thinking about like moving each individual finger.

Speaker 1

我只是想着,哦,我要捏手臂了

I'm just like, oh, I'm squeezing the arm.

Speaker 1

然后我们就能解码这个动作

And then we would be able to decode that.

Speaker 1

如果我们能解码这个动作,那将是个非常丰富的交互空间

And if we can decode that, like the that's a really rich interaction space.

Speaker 1

对吧?

Right?

Speaker 1

我不再考虑移动光标了。

I'm not thinking about moving a cursor anymore.

Speaker 1

我只想着移动我的手臂。

I'm just thinking about moving my arm.

Speaker 1

显然,因为这是我以前能做到的事,这真的非常酷。

And obviously, because this is something I could do before, that's also, like, really, really cool.

Speaker 1

这不是光标,而是一条完整的手臂,这条手臂我已经失去了移动能力。

It's not a cursor, but it's a it's a full arm, this arm that I no longer have the ability to move.

Speaker 1

结果并不理想,因为我们第一周尝试解码这个信号时,效果远不如解码光标信号——从直觉上讲这很合理,毕竟手臂动作空间大得多。

It did not pan out in the sense that we could not on the first week, like, we tried, actually decoding this, and it was just far worse than decoding the cursor, which makes sense intuitively in the sense that it's a much bigger space.

Speaker 1

但说到底,我们当时就是不确定。

But ultimately, like, we just, like, didn't know.

Speaker 1

所以,这只是一个直觉猜想。

So there was, a hunch.

Speaker 1

它没成功。

It didn't work.

Speaker 1

这并不意味着未来也不会成功。

That doesn't mean that it won't work in the future.

Speaker 1

但当时,这就像是在赌——如果我们能解码整条手臂,需要付出什么代价才能实现?又如何将之与我们已知的目标统一起来?

But at the time, like, it was just a bet on what if we could decode the full arm, what would it take to get there, and then how do we unify that with what we already know we wanna do?

Speaker 0

真有趣。

It's funny.

Speaker 0

我提前看了你们想招的第二位设计师的职位描述,其中有句话我记在了笔记里。

I went ahead and read the job description for the second designer that you're trying to bring on, and there was a line that I pasted into my notes.

Speaker 0

我想提出来找个例子说明——他们需要‘有雄心去构建超越现有可行方案局限、让人感觉神奇的事物’。

I wanted to bring it up and look for an example, which was they need the ambition to build something that feels magical beyond the local minima of that which works.

Speaker 0

我觉得你通过那段文字的挤压效果已经很好地回答了这个问题。

And I think you pretty much already answered that question with the squeezing of the text.

Speaker 0

完全没必要,但又是多么有趣的带来愉悦的方式啊。

Totally unnecessary, but also, like, what a fun way to bring delight.

Speaker 0

而你将其付诸实验的事实,正是我寻找的完美例证——即便在短期内完全不知技术上是否可行,仍选择超越极限。

And the fact that you did that into an experiment is the perfect example that I was looking for of going above and beyond even when you have no idea if this is technically even possible, at least in the short run.

Speaker 0

Ruse,这次对话真是太棒了。

Well, Ruse, this has been amazing.

Speaker 0

感谢你创造了这期最独特的节目之一。

Thank you for creating one of the most unique episodes.

Speaker 0

可能是我有史以来最独特的一期。

Probably the most unique episode that I ever had.

Speaker 0

光是听你思考这组交互和设计可能性的方式就令人着迷。

Like, just hearing the way that you think about this set of interactions and design opportunities is truly fascinating.

Speaker 0

非常感谢你今天能来与我们分享。

Really appreciate you coming on and sharing with us today.

Speaker 1

是啊。

Yeah.

Speaker 1

谢谢邀请。

Thanks for having me.

Speaker 1

这次真是太有趣了。

This was a ton of fun.

Speaker 0

在结束之前,我想花一分钟向你介绍我最喜欢的产品,因为经常有人问我用哪些工具。

Before I let you go, I wanna take just one minute to run you through my favorite products because I'm constantly asked what's in my stack.

Speaker 0

Framer是我搭建网站的工具。

Framer is how I build websites.

Speaker 0

Genway是我做研究的工具。

Genway is how I do research.

Speaker 0

Granola是我在评审时做笔记的工具。

Granola is how I take notes during crit.

Speaker 0

Jitter是我为设计制作动画的工具。

Jitter is how I animate my designs.

Speaker 0

Lovable是我用代码实现创意的工具。

Lovable is how I build my ideas in code.

Speaker 0

Maven是我寻找设计灵感的工具。

Maven is how I find design inspiration.

Speaker 0

Paper是我像创意人一样设计的工具,而Raycast则是我每一步的快捷方式。

Paper is how I design like a creative, and Raycast is my shortcut every step of the way.

Speaker 0

我精心挑选了这些公司,以便能全职制作这些节目。

Now I've hand selected these companies so that I can do these episodes full time.

Speaker 0

因此,目前支持节目的最佳方式就是去了解它们。

So by far, the number one way to support the show is to check them out.

Speaker 0

你可以在dive.club/partners找到完整列表。

You can find the full list at dive.club/partners.

关于 Bayt 播客

Bayt 提供中文+原文双语音频和字幕,帮助你打破语言障碍,轻松听懂全球优质播客。

继续浏览更多播客