Making superintelligent A.I. is a really hard challenge.

创造出一个超级慧体确实是个很大的挑。

Could you two really have some sort of superintelligent child?

你俩真的会出那种商超群的宝宝吗?

So imagine we hit upon a design of superintelligent AI that has no safety concerns.

想象我们正好做出了一个没有任何安全隐患的超级人工能。

I bet there are those of you who doubt that superintelligent AI is possible, much less inevitable.

我打赌你们中有些人还会怀疑超级人工能是否可能实现，认为我是在小题大做。

But, as a thought experiment, let's just say that superintelligent computers will emerge.

但是，作为一个思想实验，我们只说超级能计算机将会出现。

Still others view AI with more suspicion – why would a superintelligent AI waste its time taking care of us?

还有一些人对人工能抱有更的怀疑——为什么一个超级能的人工能会浪费时间来照顾我们？

Right? So imagine if we just built a superintelligent AI that was no smarter than your average team of researchers at Stanford or MIT.

那么，现在让我们来想象一下我们刚建好一个超级人工能机器，大概和斯坦福或是麻省理工学院的研究员的平均水平差不吧。

The point here is that we should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever.

所以我们不能 对我们能够永远控制一个超级能体的能力 表示过度自信。

Maybe a single, superintelligent civilization has indeed taken over the galaxy and has imposed strict radio silence because it's paranoid of any potential competitors.

可能一个非有能的文明已经占领了整个银河系，并且故意把辐射波屏蔽了，因为它们害怕潜在的竞争对手。

So, for him, if I can just summarize poorly, so for him, the first superintelligent system will just run away with everything.

所以，对他来说，如果我总结得不好，那么对他来说， 第一个超级能系统就会把一切都跑掉。

And then later in the year, Madhu and I will be back with more AI, a whole new series about superintelligent or godlike AI.

今年晚些时候， Madhu 和我将带着更人工能回来，这是一个关于超级能或神一般的人工能的全新系列。

What if the thing that we recognize as superintelligent is actually living at stage five, that the thing that's stage six, enlightenment, is not very productive?

- 如果我们认为超级能的东西实际上在第五阶段，而第六阶段的东西（启蒙运动）效率不高，该怎么办？

A superintelligent AI is by definition very good at attaining its goals, so the most important thing for us to do is to ensure that its goals are aligned with ours.

超级能很善于完成它的目标，所以对我们来说，最重要的是确保它的目标和我们的一致。

The deeper problem is that building superintelligent AI on its own seems likely to be easier than building superintelligent AI and having the completed neuroscience that allows us to seamlessly integrate our minds with it.

仅仅制造出超级人工能机器可能要比既制造超级人工能，又让其拥有能让我们的思想和超级人工能无缝对接的完整的神经科学系统要简单很。

The " Culture" series by Iain M. Banks, a Scottish novelist, goes further, considering a world in which ai has grown sufficiently powerful as to be superintelligent—operating far beyond anything now foreseeable.

苏格兰小说家伊恩·班克斯 (Iain M. Banks) 的“文化” (Culture) 系列更进一步，考虑到一个人工能已经强大到足以成为超级能的世界——其运作方式远远超出现在可预见的范围。

Even if created solely with humanity's best interests in mind, superintelligent AI could pose an existential risk if it isn't perfectly aligned with human values— a task scientists are finding extremely difficult.

即使在创造时完全考虑到人类的最大利益，如果超级能 AI 不能完全符合人类价值观，它也可能会带来存风险——科学家们发现这是一项极其困难的任务。

So you've been taken hostage by a superintelligent mass murdering cannibalistic serial killer that's hellbent on making you his main course- how in the world are you going to get out of this one?

所以你被一个超级能的大规模谋杀食人连环杀手劫持为人质，他一心要让你成为他的主菜 - 你到底要怎么摆脱这个？

Others predict humans staying largely human, but with superintelligent computers as a benevolent force, emerging as a caretaker for humanity – running all the farms, curing diseases, directing robots to pick-up trash, building new homes and many other functions.

其他人则预测，人类在很大程度上仍将是人类，但超级能计算机会成为一股仁慈的力量，成为人类的看守者——管理所有农场、治疗疾病、指挥机器人捡垃圾、建造新家园和许其他功能。