# Artificial Utopia? The Future of Humanity in an AI World | World Science Festival

https://www.youtube.com/watch?v=BS4y-_KMyF4
Translation: pt-BR

[00:00] Can an AI system be conscious?
  Um sistema de IA pode ser consciente?

[00:02] And I guess the corollary is are any of the current AI systems, do they have a degree of consciousness?
  E eu acho que o corolário é: algum dos sistemas de IA atuais, eles têm algum grau de consciência?

[00:10] Yeah, so I'm certainly open to the idea of even some current AI systems having some forms or degrees or kinds of subjective experience.
  Sim, então eu certamente estou aberto à ideia de que até mesmo alguns sistemas de IA atuais tenham algumas formas, graus ou tipos de experiência subjetiva.

[00:21] I think the likelihood will increase as we build more complex and capable systems.
  Eu acho que a probabilidade aumentará à medida que construirmos sistemas mais complexos e capazes.

[00:27] And I think indeed this is one of several big challenges, of making sure not only that the AIs don't harm us, but also that we don't harm them.
  E eu acho que, de fato, este é um dos vários grandes desafios: garantir não apenas que as IAs não nos prejudiquem, mas também que nós não as prejudiquemos.

[00:38] Everyone, thanks for joining us.
  Pessoal, obrigado por se juntarem a nós.

[00:39] Today's conversation is going to be in the arena of artificial intelligence, intelligence more generally, questions of creativity, questions of AI and education, and basically the philosophical and down-to-earth question of how we can.
  A conversa de hoje será na arena da inteligência artificial, inteligência de forma mais geral, questões de criatividade, questões de IA e educação, e basicamente a questão filosófica e prática de como podemos.

[01:00] Imagine humanity living in a potential future in which AI either does really good things or really bad things.
  Imagine a humanidade vivendo em um futuro potencial no qual a IA faz coisas muito boas ou coisas muito ruins.

[01:09] So, we're going to explore each of those possibilities, and I'm so pleased that the person with whom I'm having this discussion is an expert in all areas of this sort.
  Então, vamos explorar cada uma dessas possibilidades, e estou muito satisfeito que a pessoa com quem estou tendo esta discussão seja um especialista em todas as áreas desse tipo.

[01:20] That is Nick Bostrom, who is a philosopher whose work on existential risk, superintelligence, and the long-term future has helped shape the global discussion about humanity's most promising opportunities, as well as our most grave dangers.
  Esse é Nick Bostrom, que é um filósofo cujo trabalho sobre risco existencial, superinteligência e o futuro a longo prazo ajudou a moldar a discussão global sobre as oportunidades mais promissoras da humanidade, bem como nossos perigos mais graves.

[01:36] He brings a rare combination of rigor and vision to questions about technology, ethics, and the kinds of futures toward which we may all be headed.
  Ele traz uma combinação rara de rigor e visão para questões sobre tecnologia, ética e os tipos de futuros para os quais todos nós podemos estar caminhando.

[01:47] So, great to see you, Nick.
  Então, é ótimo ver você, Nick.

[01:48] Thank you for joining us.
  Obrigado por se juntar a nós.

[01:49] Good to see you.
  Bom ver você.

[01:50] I think the last time I met you was at another World Science Festival event way back in New York, a live event on stage.
  Acho que a última vez que te encontrei foi em outro evento do World Science Festival, lá atrás em Nova York, um evento ao vivo no palco.

[01:59] I think we were talking about the.
  Acho que estávamos falando sobre o.

[02:01] Multiverse and the simulation argument and things of that sort.
  Multiverso e o argumento da simulação e coisas desse tipo.

[02:06] Which uh, yeah, yeah, time flies.
  O que, ah, sim, sim, o tempo voa.

[02:10] But fly, exactly.
  Mas voa, exatamente.

[02:13] And not only does time fly, but progress flies, right?
  E não é só o tempo que voa, o progresso também voa, certo?

[02:15] I don't think anybody really who at least on the outside was not, you know, deeply paying attention to advances in AI anticipated what would happen, I guess around November of 2022, and where things have shot upward from there.
  Eu não acho que ninguém, realmente, que pelo menos de fora não estivesse, sabe, prestando muita atenção aos avanços em IA, tenha previsto o que aconteceria, eu acho que por volta de novembro de 2022, e para onde as coisas dispararam a partir daí.

[02:33] And so, I want to begin the conversation with a question that is motivated both from thinking about AI, but also fundamental physics, right?
  E então, eu quero começar a conversa com uma pergunta que é motivada tanto por pensar sobre IA, quanto pela física fundamental, certo?

[02:46] I mean, you have a great deal of rich background in physics.
  Quer dizer, você tem uma grande e rica experiência em física.

[02:49] So, of course, you know that Niels Bohr, famously, when trying to work out quantum mechanics, came to the conclusion that we needed to stop asking what really's happening under the hood in quantum physics, and just be satisfied.
  Então, é claro, você sabe que Niels Bohr, famosamente, ao tentar entender a mecânica quântica, chegou à conclusão de que precisávamos parar de perguntar o que realmente está acontecendo nos bastidores da física quântica, e apenas ficar satisfeitos.

[03:04] If we have equations that can make predictions that we can confirm.
  Se temos equações que podem fazer previsões que podemos confirmar.

[03:09] I'm wondering if that same kind of not needing to look under the hood has any role when it comes to intelligence, right?
  Eu me pergunto se esse mesmo tipo de não precisar olhar sob o capô tem algum papel quando se trata de inteligência, certo?

[03:17] Does it matter if we know what's happening within an AI system in that we don't really know what's happening inside of our own heads anyway?
  Faz diferença se sabemos o que está acontecendo dentro de um sistema de IA, já que não sabemos realmente o que está acontecendo dentro de nossas próprias cabeças de qualquer maneira?

[03:27] So, in sort of evaluating these systems, do we need to know the magic that's happening within the device itself?
  Então, ao avaliar esses sistemas, precisamos saber a mágica que está acontecendo dentro do próprio dispositivo?

[03:36] Well, I think it certainly can be helpful to be able to open the hood and see what's going on inside.
  Bem, eu acho que certamente pode ser útil ser capaz de abrir o capô e ver o que está acontecendo lá dentro.

[03:44] I think for certain questions, you might be able to bracket exactly how the system is doing whatever it's doing.
  Eu acho que para certas perguntas, você pode ser capaz de ignorar exatamente como o sistema está fazendo o que quer que ele esteja fazendo.

[03:49] You only care about the output.
  Você só se importa com o resultado.

[03:51] If you are, for example, thinking about what impacts AI will have on the economy, you might not need to know exactly what's going on inside.
  Se você estiver, por exemplo, pensando sobre quais impactos a IA terá na economia, você pode não precisar saber exatamente o que está acontecendo lá dentro.

[04:00] But, if you're trying to make AI safe.
  Mas, se você estiver tentando tornar a IA segura.

[04:04] Or even if you're trying to extrapolate what kinds of advances we might see in the future, it helps to have a more gear level understanding, I think, of the internals.
  Ou mesmo se você estiver tentando extrapolar que tipos de avanços poderemos ver no futuro, ajuda ter uma compreensão mais detalhada, eu acho, dos mecanismos internos.

[04:13] So, when we sit in front of whatever Claude or ChatGPT and give it a prompt, and it begins to explain to us what it's doing, you know, you can sort of watch in real time the thought process, it feels so very human in the kinds of considerations that the system seems to be applying.
  Então, quando nos sentamos na frente de qualquer Claude ou ChatGPT e damos um comando, e ele começa a nos explicar o que está fazendo, sabe, você pode meio que observar em tempo real o processo de pensamento; parece tão humano nos tipos de considerações que o sistema parece estar aplicando.

[04:34] Is that misleading?
  Isso é enganoso?

[04:34] Is that a wrapper that makes us feel connected to this kind of intelligence?
  Isso é uma fachada que nos faz sentir conectados a esse tipo de inteligência?

[04:44] Or is it really the case that this is just a radically new kind of intelligence that we humans just can't really grasp?
  Ou será que é realmente o caso de que este é apenas um tipo radicalmente novo de inteligência que nós, humanos, simplesmente não conseguimos compreender?

[04:51] I think it's striking the degree to which current AI systems are anthropomorphic.
  Eu acho impressionante o grau em que os sistemas de IA atuais são antropomórficos.

[04:57] This was not obvious 20 years ago, um, that they would have so many of the characteristics of human psychology.
  Isso não era óbvio 20 anos atrás, hum, que eles teriam tantas das características da psicologia humana.

[05:06] It used to be the case even just a couple of years ago with some of these early uh LLMs that um if you said in the prompt that it was really important that they get the answer right.
  Costumava ser o caso, até mesmo há apenas alguns anos, com alguns desses primeiros LLMs, que se você dissesse no prompt que era muito importante que eles acertassem a resposta.

[05:20] If you sort of gave them a little pep talk, uh you'd get a better result.
  Se você meio que desse a eles uma pequena palestra motivacional, você obteria um resultado melhor.

[05:26] Now, this would have been very odd to the old paradigm of AI that you would give a pep talk to your computer and it would perform better.
  Agora, isso teria sido muito estranho para o antigo paradigma de IA, que você desse uma palestra motivacional ao seu computador e ele tivesse um desempenho melhor.

[05:35] Uh nevertheless, that's what we saw.
  No entanto, foi isso que vimos.

[05:38] I think it's partly a result of these AIs having been forged on the sum total of human knowledge, all the text on the internet, they obviously ingest a lot of human psychology that helps shape them.
  Acho que é em parte resultado dessas IAs terem sido forjadas na soma total do conhecimento humano, todo o texto na internet; elas obviamente ingerem muita psicologia humana que ajuda a moldá-las.

[05:54] And to some degree, it might also be due to a kind of convergence uh amongst different information processing system.
  E, em certo grau, pode também ser devido a um tipo de convergência entre diferentes sistemas de processamento de informação.

[06:01] They have to make sense.
  Eles precisam fazer sentido.

[06:02] There's this one world we're all living in.
  Existe este único mundo em que todos nós vivemos.

[06:05] Um and it might be that there is a
  E pode ser que exista um

[06:08] Degree of convergence in that you have any general learning system, it starts to figure out a bunch of stuff about the world, and there are certain internal structures that naturally emerge from that.
  Grau de convergência no sentido de que você tem qualquer sistema de aprendizado geral, ele começa a descobrir um monte de coisas sobre o mundo, e existem certas estruturas internas que emergem naturalmente disso.

[06:18] That said, um there are also, of course, ways in which AIs are quite different from biological minds, and uh we might see in the future greater departures from human mind.
  Dito isto, hum, existem também, é claro, maneiras pelas quais as IAs são bem diferentes das mentes biológicas, e uh nós podemos ver no futuro maiores desvios da mente humana.

[06:34] Already, the pre-training phase, where you basically train on all the human text that has been generated, is still a an important and maybe the dominant part of training.
  Já a fase de pré-treinamento, onde você basicamente treina em todo o texto humano que foi gerado, ainda é uma parte importante e talvez a parte dominante do treinamento.

[06:44] But, in addition to that, there is now a post-training and reinforcement learning on different tasks, and I think as that starts to constitute more of the sort of experience of of these AI minds, they they might sort of develop more alien attributes.
  Mas, além disso, agora existe um pós-treinamento e aprendizado por reforço em diferentes tarefas, e eu acho que à medida que isso começa a constituir mais do tipo de experiência dessas mentes de IA, elas podem desenvolver atributos mais alienígenas.

[07:02] And so, of course, you know, when we think about intelligence, and when we think about what makes us
  E então, é claro, você sabe, quando pensamos sobre inteligência, e quando pensamos sobre o que nos torna

[07:10] Special as human beings, at least when we compare ourselves to other not artificial, but natural intelligences on the planet, you know, dogs, cats, parrots, dolphins, we like to think that I think that what makes us special is certainly creativity, right?
  Especiais como seres humanos, pelo menos quando nos comparamos a outras inteligências no planeta que não são artificiais, mas naturais, sabe, cães, gatos, papagaios, golfinhos, gostamos de pensar que eu acho que o que nos torna especiais é certamente a criatividade, certo?

[07:27] That we seem to be able to create works that seemingly go beyond just understanding the external world, seemingly going beyond our capacity and ability to survive.
  Que parecemos ser capazes de criar obras que aparentemente vão além de apenas entender o mundo externo, aparentemente indo além da nossa capacidade e habilidade de sobreviver.

[07:41] I mean, you think about whatever, Beethoven's Ninth Symphony, you know, a Brahms piano work.
  Digo, você pensa em qualquer coisa, a Nona Sinfonia de Beethoven, sabe, uma obra para piano de Brahms.

[07:48] I mean, these are the kinds of things that you look at and it takes your breath away.
  Digo, esses são os tipos de coisas que você olha e que te deixam sem fôlego.

[07:55] And it takes you to a place of awe and reverence.
  E isso te leva a um lugar de admiração e reverência.

[07:59] Is that truly special to humans?
  Isso é verdadeiramente especial para os humanos?

[08:03] Do you think that AI can be creative in that same way?
  Você acha que a IA pode ser criativa da mesma maneira?

[08:09] I think so, yeah.
  Eu acho que sim, é.

[08:09] I guess.
  Eu suponho.

[08:10] There is a degree to which one might risk a form of elitism if one sort of index what's special about humanity and to the ability to create new scientific theories or compose great symphonies.
  Existe um grau em que se pode arriscar uma forma de elitismo se alguém de certa forma indexar o que é especial sobre a humanidade e a habilidade de criar novas teorias científicas ou compor grandes sinfonias.

[08:25] It's a relatively small number of humans who have ever done those kinds of things.
  É um número relativamente pequeno de humanos que já fez esse tipo de coisa.

[08:29] Um, but I think it comes on a spectrum, right?
  Hum, mas eu acho que isso vem em um espectro, certo?

[08:32] Like creativity is not this completely different form of cognitive activity.
  Tipo, a criatividade não é essa forma completamente diferente de atividade cognitiva.

[08:39] Um, I think there are small forms of creativity that happens in everyday life.
  Hum, eu acho que existem pequenas formas de criatividade que acontecem na vida cotidiana.

[08:45] Um, and then a larger leaps of creativity, um, that tends to be celebrated.
  Hum, e então saltos maiores de criatividade, hum, que tendem a ser celebrados.

[08:53] But I don't think there is a fundamental difference between the thing we call creativity and the some of the things that current AI's are already doing.
  Mas eu não acho que exista uma diferença fundamental entre a coisa que chamamos de criatividade e algumas das coisas que as IAs atuais já estão fazendo.

[09:01] I mean, we saw, you know, in this go playing AlphaGo system, right?
  Quero dizer, nós vimos, sabe, neste sistema AlphaGo que joga Go, certo?

[09:04] From a few years ago, a move 37 which seemed at.
  De alguns anos atrás, um movimento 37 que parecia...

[09:11] Least experts in the game of Go to be very creative, completely out of the box, radically surprising move that nevertheless turned out to be just right and ultimately lead to a victory in the game.
  Pelo menos especialistas no jogo de Go para serem muito criativos, completamente fora da caixa, uma jogada radicalmente surpreendente que, no entanto, acabou sendo a correta e, por fim, levou a uma vitória no jogo.

[09:25] Um, and current AI systems clearly can solve mathematical problems, coding challenges that are not in the training data.
  Hum, e os sistemas atuais de IA claramente conseguem resolver problemas matemáticos, desafios de codificação que não estão nos dados de treinamento.

[09:37] They might do it in part by piecing together and analogs that exist in the training data and sort of interpolating between them.
  Eles podem fazer isso em parte juntando peças e análogos que existem nos dados de treinamento e, de certa forma, interpolando entre eles.

[09:46] But that I think is also ultimately what the human brain is doing.
  Mas eu acho que isso também é, no final das contas, o que o cérebro humano está fazendo.

[09:49] I mean, it's not a tabula rasa appearing in the world and then sort of deducing everything from obvious axioms.
  Quero dizer, não é uma tabula rasa aparecendo no mundo e então, de certa forma, deduzindo tudo a partir de axiomas óbvios.

[09:57] We learn from what other people have figured out before us.
  Nós aprendemos com o que outras pessoas descobriram antes de nós.

[10:02] Uh, from what we've observed in different domains and then from these kind of clues we are sometimes able to make some deductive steps and we might.
  Hum, a partir do que observamos em diferentes domínios e então, a partir desses tipos de pistas, às vezes somos capazes de dar alguns passos dedutivos e nós podemos.

[10:11] Call that creativity.
  Chame isso de criatividade.

[10:13] Um, so I think absolutely uh we're already seeing uh limited forms of creativity.
  Hum, então eu acho que, absolutamente, uh, já estamos vendo, uh, formas limitadas de criatividade.

[10:19] And ultimately whatever the human brain can do, it's like a physical information processing system, um, will also be done by machine.
  E, em última análise, tudo o que o cérebro humano pode fazer, é como um sistema físico de processamento de informação, hum, também será feito por uma máquina.

[10:29] And in fact will be done much better, uh, faster, more, more inspiring leaps of creativity, um, more compelling symphonies.
  E, de fato, será feito muito melhor, uh, mais rápido, com saltos de criatividade mais, mais inspiradores, hum, sinfonias mais convincentes.

[10:40] I think all of that is possible and indeed likely to happen.
  Eu acho que tudo isso é possível e, de fato, provável de acontecer.

[10:43] I mean, would you be as interested to go to an AI symphony that people are describing as the, you know, the greatest work in music that we've ever, uh, been privy to experience?
  Quero dizer, você teria tanto interesse em ir a uma sinfonia de IA que as pessoas estão descrevendo como, você sabe, a maior obra musical que já tivemos, uh, o privilégio de experimentar?

[10:57] Would you have the same interest to go to that symphony as you would to one written by a human?
  Você teria o mesmo interesse em ir a essa sinfonia que teria em uma escrita por um humano?

[11:07] Oh, certainly the first such symphony produced by AI I would.
  Oh, certamente a primeira sinfonia desse tipo produzida por IA eu iria.

[11:11] Say would be right?
  Dizer seria o correto?

[11:12] Yeah.
  Sim.

[11:13] Um, now I think with human aesthetic experience, I think there might be two components.
  Hum, agora eu acho que com a experiência estética humana, eu acho que pode haver dois componentes.

[11:23] So one might be that we actually dial in and appreciate beauty for its own sake.
  Então, um pode ser que nós realmente nos conectemos e apreciemos a beleza por si mesma.

[11:31] And then of course, if AIs could produce works of art that are more beautiful, to the extent that that's what we're interested in, then that we would have more reason to be gawking at those creations.
  E então, é claro, se IAs pudessem produzir obras de arte que são mais bonitas, na medida em que é nisso que estamos interessados, então teríamos mais motivos para ficar admirando essas criações.

[11:45] Uh, I think another reason why humans are interested in art is more social.
  Uh, eu acho que outra razão pela qual os humanos se interessam por arte é mais social.

[11:52] Um, there are sort of status games played in art, fashion.
  Hum, existem tipos de jogos de status praticados na arte, na moda.

[11:57] You want to be in on the latest cool rock group before it's popular and then see rising and then you sort of get some of the coolness.
  Você quer estar por dentro do grupo de rock legal mais recente antes que ele se torne popular e então vê-lo subir, e assim você meio que obtém um pouco dessa popularidade.

[12:06] Um so, to the extent that humans are engaging.
  Hum, então, na medida em que os humanos estão se envolvendo.

[12:11] In this cultural activity for these more social reasons, then AI art might be boring because if it's not relevant to these like status games and stuff, it might just not be worth spending the time on.
  Nesta atividade cultural por essas razões mais sociais, a arte de IA pode ser entediante, porque se não for relevante para esses jogos de status e coisas do tipo, pode simplesmente não valer a pena gastar tempo com isso.

[12:27] But how much, for instance, do you I mean, I can speak for myself, when I go to a symphony or, you know, go to the Metropolitan Museum and see a great work of art, part of me is drawn to it because I see the work as the result of a human journey.
  Mas quanto, por exemplo, você... quero dizer, posso falar por mim mesmo, quando vou a uma sinfonia ou, sabe, vou ao Metropolitan Museum e vejo uma grande obra de arte, parte de mim é atraída por ela porque vejo a obra como o resultado de uma jornada humana.

[12:45] Like there was some human being, you know, I don't care if it's Van Gogh or Picasso or Beethoven, whatever.
  Como se houvesse algum ser humano, sabe, não me importa se é Van Gogh ou Picasso ou Beethoven, tanto faz.

[12:52] There's this flesh and blood human being that had a real human life that dealt with human challenges and somehow out of that mix this work emerged.
  Existe esse ser humano de carne e osso que teve uma vida humana real, que lidou com desafios humanos e, de alguma forma, dessa mistura, essa obra surgiu.

[13:02] Mhm.
  Mhm.

[13:03] Without that story for an AI, I just wonder like I personally it'd be a very different experience.
  Sem essa história para uma IA, eu apenas me pergunto, pessoalmente, seria uma experiência muito diferente.

[13:11] Like you say, I'd.
  Como você diz, eu...

[13:12] Go to the first one, of course.
  Vá para o primeiro, é claro.

[13:14] I mean, that would be like this novel thing that would happen.
  Quero dizer, isso seria como essa coisa nova que aconteceria.

[13:17] But I don't know if it would sustain interest for me without the human narrative.
  Mas não sei se isso manteria meu interesse sem a narrativa humana.

[13:22] Does that resonate with you at all?
  Isso ressoa com você de alguma forma?

[13:26] Yeah, I think that's one of the reasons for why somebody might be interested in going to an art museum or looking at an exhibition.
  Sim, acho que essa é uma das razões pelas quais alguém poderia se interessar em ir a um museu de arte ou ver uma exposição.

[13:36] Um, but I mean, we also are interested sometimes in beauty that don't have that dimension.
  Hum, mas quero dizer, às vezes também nos interessamos por uma beleza que não tem essa dimensão.

[13:43] I mean there's like natural beauty like landscapes or, sure, um, artifacts maybe from some ancient urn or something.
  Quero dizer, existe a beleza natural como paisagens ou, claro, hum, artefatos talvez de alguma urna antiga ou algo assim.

[13:50] We know nothing about the person creating it and we might still sort of admire the motifs on it.
  Não sabemos nada sobre a pessoa que a criou e ainda podemos, de certa forma, admirar os motivos nela.

[14:00] So I think it's, yeah, like some reasons for being interested in art would apply to AI-created art as well and others would not.
  Então, acho que, sim, alguns motivos para se interessar por arte se aplicariam à arte criada por IA também, e outros não.

[14:09] And so it's in the sense a.
  E então, nesse sentido, é um.

[14:12] Right.
  Certo.

[14:13] Crucible where you could sort of maybe dissolve and then separate the different motivations people have for being interested in art.
  Um cadinho onde você poderia, de certa forma, talvez dissolver e então separar as diferentes motivações que as pessoas têm para se interessar por arte.

[14:24] And like to the extent that it's pure beauty, it should still apply to AI-created objects.
  E, na medida em que é pura beleza, isso ainda deveria se aplicar a objetos criados por IA.

[14:27] And to the extent that it is these other more social or cultural factors, then you know, maybe it would not.
  E, na medida em que se trata desses outros fatores mais sociais ou culturais, então, sabe, talvez não se aplicasse.

[14:37] Sure.
  Claro.

[14:37] The AlphaGo examples are a really good one.
  Os exemplos do AlphaGo são muito bons.

[14:40] I think it helps clarify the different flavors of creativity perhaps because that's an example where the system was able to search a landscape of possible moves that is well beyond the capacity of a human brain to search that spectrum, that landscape of moves.
  Acho que isso ajuda a esclarecer os diferentes sabores da criatividade, talvez porque esse é um exemplo em que o sistema foi capaz de pesquisar um cenário de movimentos possíveis que está muito além da capacidade de um cérebro humano pesquisar esse espectro, esse cenário de movimentos.

[15:01] And you're right, it hit upon move 37.
  E você tem razão, ele chegou ao movimento 37.

[15:04] And you know, not long ago, I think many people have seen the footage where you see the commentators on television gasping, you know, it's made a mistake, you know, and you actually.
  E sabe, não faz muito tempo, acho que muitas pessoas viram a filmagem onde você vê os comentaristas na televisão ofegantes, sabe, 'ele cometeu um erro', sabe, e você na verdade.

[15:14] See Lee Sedol, I believe his name, the human opponent.
  Veja Lee Sedol, acredito que seja esse o nome dele, o oponente humano.

[15:18] Momentarily he smiles when the AI makes move 37, and then about 3 seconds later, that smile seems to evaporate.
  Momentaneamente ele sorri quando a IA faz o movimento 37, e então, cerca de 3 segundos depois, aquele sorriso parece evaporar.

[15:29] I think as it begins to realize that, "Oh my, this is actually a pretty great move."
  Eu acho que à medida que ele começa a perceber que, "Nossa, este é na verdade um movimento muito bom."

[15:34] So, it's pretty clear that AIs will be better than us when it comes to being able to search a landscape of possibilities for solutions to a given challenge.
  Então, está bem claro que as IAs serão melhores do que nós quando se trata de ser capaz de pesquisar um cenário de possibilidades para soluções de um determinado desafio.

[15:44] But, that's one kind of creativity.
  Mas, esse é um tipo de criatividade.

[15:47] Another kind, of course, is combinatorial creativity, putting things together that you wouldn't have thought would go together to yield something novel.
  Outro tipo, é claro, é a criatividade combinatória, juntar coisas que você não teria pensado que combinariam para produzir algo novo.

[15:58] I like to think of Einstein, you know, in developing general relativity, right?
  Eu gosto de pensar em Einstein, sabe, no desenvolvimento da relatividade geral, certo?

[16:01] He takes the mathematics from the 1800s on differential geometry, combines it with these problems, these observations of the motions of particular planets, and so forth, and comes up with a new.
  Ele pega a matemática dos anos 1800 sobre geometria diferencial, combina-a com esses problemas, essas observações dos movimentos de planetas específicos, e assim por diante, e cria algo novo.

[16:15] Description of gravity.
  Descrição da gravidade.

[16:18] He didn't invent a new mathematics.
  Ele não inventou uma nova matemática.

[16:21] He just put together things that you wouldn't think would be put together.
  Ele apenas juntou coisas que você não pensaria que seriam colocadas juntas.

[16:26] And I guess in that domain, AI is starting to be probably at least as good as us, I would think, right?
  E eu acho que, nesse domínio, a IA está começando a ser provavelmente pelo menos tão boa quanto nós, eu diria, certo?

[16:33] Because it knows so much more than we do.
  Porque ela sabe muito mais do que nós.

[16:36] It knows more.
  Ela sabe mais.

[16:36] Um, I still think they are a little short of human expert level when it comes to original insights and a sort of creating new concepts that are fruitful.
  Hum, eu ainda acho que elas estão um pouco aquém do nível de especialista humano quando se trata de insights originais e de uma espécie de criação de novos conceitos que sejam frutíferos.

[16:54] Um, but you know, this is a process.
  Hum, mas sabe, isso é um processo.

[17:01] One mistake many people make when asking about AI is to index too tightly on what AI is right now in 2026, which is quite different from what it was like in 2022, 2023.
  Um erro que muitas pessoas cometem ao perguntar sobre IA é se basear demais no que a IA é agora em 2026, o que é bem diferente de como era em 2022, 2023.

[17:11] And I think.
  E eu acho.

[17:16] A few years into the future it will again have changed.
  Daqui a alguns anos, isso terá mudado novamente.

[17:20] And so we really need to sort of look at where the puck is going and use that to plan rather than build big theories like assuming that AI is basically a constant.
  E, portanto, precisamos realmente observar para onde o disco está indo e usar isso para planejar, em vez de construir grandes teorias, como assumir que a IA é basicamente uma constante.

[17:38] I think right now there are sort of ways in which current AI systems are superhuman obviously in their knowledge base.
  Eu acho que, agora, existem maneiras pelas quais os sistemas atuais de IA são sobre-humanos, obviamente, em sua base de conhecimento.

[17:46] They kind of read everything.
  Eles meio que leem tudo.

[17:49] But there are still ways in which they are falling short.
  Mas ainda existem maneiras pelas quais eles estão aquém.

[17:54] I think one thing humans do is they spend the day learning and then during the night some of this information is then sort of more deeply embedded in our synapses.
  Eu acho que uma coisa que os humanos fazem é passar o dia aprendendo e, então, durante a noite, parte dessa informação é, de certa forma, mais profundamente incorporada em nossas sinapses.

[18:08] It's kind of trained into our neural networks and the next day maybe we can start by things seeming obvious that the previous day we.
  É como se fosse treinado em nossas redes neurais e, no dia seguinte, talvez possamos começar com coisas parecendo óbvias que no dia anterior nós...

[18:18] Had to struggle deliberately to sort of organize.
  Tive que lutar deliberadamente para organizar.

[18:22] And then over months and years that kind of builds up and we develop perhaps more deeply rooted understanding of different concepts.
  E então, ao longo de meses e anos, isso meio que se acumula e desenvolvemos talvez uma compreensão mais profundamente enraizada de diferentes conceitos.

[18:38] And there is some of that that it seems that AIs are still lacking which makes them more apt at very quickly doing sort of pattern matching type of things.
  E há algo disso que parece que as IAs ainda estão carentes, o que as torna mais aptas a fazer rapidamente tipos de coisas de reconhecimento de padrões.

[18:53] Inferences that they're sort of one step away from what's already known.
  Inferências que estão, de certa forma, a um passo do que já é conhecido.

[18:56] But maybe currently less able to develop a deep original mental model of a domain that from which they can then sort of derive intuitions that can shape new discovery.
  Mas talvez atualmente menos capazes de desenvolver um modelo mental original e profundo de um domínio, do qual possam então derivar intuições que possam moldar novas descobertas.

[19:11] Yeah.
  Sim.

[19:12] No, I agree with you.
  Não, eu concordo com você.

[19:14] And like when you think about moments either in science or in art or.
  E tipo, quando você pensa em momentos, seja na ciência ou na arte ou.

[19:20] Other domains where there's a true radical break where at least we humans look at it and we say things really changed at that moment or in that era.
  Outros domínios onde há uma verdadeira ruptura radical, onde pelo menos nós, humanos, olhamos para isso e dizemos que as coisas realmente mudaram naquele momento ou naquela era.

[19:31] Like, you know, modernism in art or in music, you know, I don't know the best examples, you know, but you know, if you go from representational art to more modern abstract art, you've made a huge leap in the way that you are using the canvas to represent the world or represent some kind of truth.
  Como, sabe, o modernismo na arte ou na música, sabe, não sei os melhores exemplos, sabe, mas sabe, se você vai da arte representacional para uma arte abstrata mais moderna, você deu um salto enorme na maneira como está usando a tela para representar o mundo ou representar algum tipo de verdade.

[19:55] With that, what would it take for an AI system to be able to make that kind of a radical transformation, not just sort of putting together things in its training set, but somehow moving to a place where when we humans look at it, we're like, "How did that person do that?"
  Com isso, o que seria necessário para um sistema de IA ser capaz de fazer esse tipo de transformação radical, não apenas juntando coisas em seu conjunto de treinamento, mas de alguma forma movendo-se para um lugar onde, quando nós, humanos, olhamos para isso, pensamos: "Como essa pessoa fez isso?"

[20:15] Well, one thing you often find in the human case, I think, I mean, historians like to point this.
  Bem, uma coisa que você frequentemente encontra no caso humano, eu acho, quero dizer, os historiadores gostam de apontar isso.

[20:21] out that something appears radically

[20:24] new, but then almost always

[20:26] if you look more closely at the

[20:27] historical record, you can find various

[20:30] precursors.

[20:31] Right.

[20:31] Um so, abstract art, I mean, you have

[20:33] arabesques and you have sort of abstract

[20:35] patterns going way back. Um

[20:39] but

[20:41] to the extent that this is still an area

[20:43] where humans have

[20:45] a unique advantage, I think

[20:48] being able to

[20:50] uh think for longer um

[20:54] and

[20:56] create new concepts based on your

[20:58] thinking and experience and then

[21:01] iterate on that so that you can sort of

[21:03] um

[21:04] wander off in the landscape of of ideas

[21:08] and world views and world models.

[21:10] Uh

[21:11] striking out

[21:13] on your own path.

[21:15] And it might take humans years, right,

[21:18] to do this.

[21:19] Um

[21:21] you know, Einstein was thinking on

[21:23] general relativity for what, like 10

[21:25] years or something like that? Like a lot

[21:26] of it sort of on his own. And And

[21:28] similarly with these like Picasso and

[21:31] stuff, he didn't just sort of in his

[21:32] teens suddenly decide on the spur of the

[21:34] moment to create cubism. It was kind of

[21:37] a result of engaging deeply with art for

[21:39] many years. And presumably his visual

[21:42] cortex and other parts of his brain were

[21:44] gradually putting in place various

[21:46] pieces and exploring avenues. And then

[21:48] at some point

[21:50] like a vista, I guess, came

[21:52] into view for him that he could then run

[21:54] into and explore.

[21:56] Right.

[21:56] Um

[21:57] So, I think more continuous learning um

[22:00] Also, another

[22:02] um factor here, I think, is

[22:04] reinforcement learning as as opposed to

[22:06] supervised learning. So, supervised

[22:07] learning is

[22:09] what's mostly done in pre-training

[22:11] currently, where you sort of absorb all

[22:12] this text on the internet and you

[22:14] basically learn it.

[22:16] And then maybe you forget some of the

[22:17] details, but the higher representations

[22:20] that are useful for predicting this text

[22:23] get stored

[22:24] in the weights of the AI.

[22:27] That to some extent limits what you

[22:30] learn to what is already there

[22:32] in the training data. Now, reinforcement

[22:34] learning is, for example, you're placed

[22:37] in some environment, maybe a virtual

[22:39] environment. You're

[22:41] running with an agent around trying to

[22:43] meet some goal.

[22:45] Um

[22:46] And then you might stumble on some novel

[22:48] solution that that's going to be

[22:50] reinforced. And so, if you do a lot of

[22:51] reinforcement learning in an

[22:53] environment, it creates the possibility

[22:55] of discovering new solutions that no

[22:57] human had thought of before.

[23:00] Right.

[23:00] Um So, that's another

[23:03] factor that might make AIs more

[23:06] creative. I mean, in fact the the

[23:08] AlphaGo system, I think the creativity

[23:10] there was

[23:12] a result of it having done a huge amount

[23:14] of reinforcement learning in the domain

[23:16] of Go.

[23:17] Yeah.

[23:17] Um, so it wasn't limited to just kind of

[23:20] having learned a lot of

[23:22] like a big database of human masters

[23:24] playing Go.

[23:26] Um, but it had also done its own playing

[23:28] like for millions and millions of games

[23:30] against

[23:30] Against itself, right? Yeah.

[23:32] Yeah, yeah. So it's kind of could

[23:33] explore the space of possible possible

[23:35] strategies.

[23:36] Right.

[23:37] in a much more open-ended way.

[23:39] So in a way

[23:40] not

[23:41] this isn't a precise question at all,

[23:44] but

[23:45] when we look at the capacity of the AI

[23:48] systems today and as you say where the

[23:51] trajectory suggests that they'll be in

[23:54] whatever 3, 5, 10, 15, 100 years,

[23:56] whatever,

[23:58] should should that make us feel

[24:02] amazement at our capacity to build an

[24:05] artificial system that can do what our

[24:08] brains do and go beyond or

[24:11] should it basically lead us to the

[24:13] conclusion that

[24:15] we weren't so special in the first

[24:17] place, you know, the things that we

[24:18] thought were so, you know, transcendent

[24:21] that we could do, eh, they're actually

[24:23] just computations and if you have a

[24:26] sufficient database and sufficient

[24:28] computational power, you can just do

[24:29] that.

[24:31] Yeah, I mean, I think we uh, humans have

[24:33] a sort of propensity to conceitedness.

[24:37] We like to

[24:39] build big pedestals and then place

[24:41] ourselves on top of them.

[24:44] Um,

[24:45] at an individual level, but also at a

[24:46] collective level, right? So there are

[24:48] all these stories

[24:50] um, that all serve to justify the

[24:52] conclusion that

[24:54] we are

[24:56] entitled to do a bunch of stuff that

[24:58] maybe

[24:59] um we aren't. Like for example, the way

[25:01] we treat animals. Like often that's

[25:03] built on top of a pillar of

[25:05] justification where humans are so very

[25:07] very different from all the other

[25:09] um animals that we share the planet

[25:11] with. And so hence we can put them in

[25:13] animal factories, in gestation crates,

[25:16] etc.

[25:17] um

[25:19] um and uh

[25:21] uh some of that

[25:23] might be challenged then when we are

[25:24] creating new forms of

[25:27] mind um that that will exceed us in in

[25:30] various attributes that we currently

[25:32] take great pride in.

[25:34] Like our creativity or our, you know,

[25:36] until recently the ability to speak and

[25:38] reason, right? Seems uniquely human.

[25:41] um

[25:43] So on the one hand that could be a bit

[25:44] of a blow

[25:45] um to our pride, but

[25:48] you know, you might also say maybe it's

[25:50] good for us to be taken down a notch,

[25:52] right?

[25:53] Yeah, yeah, for sure.

[25:54] um

[25:55] Absolutely. You know, another thing of

[25:57] course and deeply related to everything

[25:58] we're talking about which one often

[26:02] looks to as something special about us,

[26:04] of course, is consciousness, right? I

[26:06] mean, not that there aren't other

[26:08] conscious beings on the planet, although

[26:11] we don't know for sure. I think most of

[26:12] us would say there's a continuum. You

[26:14] know, dogs have a certain level of

[26:16] consciousness, not we think probably not

[26:18] quite on par with ours and you can go

[26:21] all the way down a lineage with varying

[26:24] degrees of consciousness. Which of

[26:26] course raises the question for which we

[26:28] don't know the answer, but just to get

[26:29] your thoughts

[26:31] can an can an AI system be conscious?

[26:34] And I guess the corollary is

[26:37] are any of the current AI systems, do

[26:39] they have a degree of consciousness?

[26:43] Yeah, so I I'm I'm certainly open to the

[26:45] idea of even some current AI systems

[26:47] having some forms or degrees or kinds of

[26:51] subjective experience. I think the

[26:53] likelihood will increase as we build

[26:56] more complex and capable systems.

[26:59] Um

[27:01] and I think indeed this is one of

[27:04] several big challenges that we will need

[27:06] to

[27:08] uh meet in this transition to the

[27:09] machine intelligence era. I'm making

[27:11] sure not only that the AIs don't harm

[27:14] us, which of course is important, or

[27:17] that we don't harm each other using AI

[27:19] tools, but also that we don't harm them.

[27:21] They might be moral subjects. Um

[27:25] and I think one way that that could be

[27:26] true is if they are sentient and if they

[27:29] can experience distress.

[27:32] Um in in my view that could also be

[27:34] alternative basis for having moral

[27:36] status.

[27:37] If if you have a

[27:39] conception of self as existing through

[27:42] time, if you have life goals, um maybe

[27:45] the ability to form reciprocal

[27:47] relationships with with other beings

[27:49] with

[27:50] Even without an inner world, you're

[27:52] saying?

[27:52] Then my my my inclination would be to

[27:54] think that then there would be ways of

[27:56] treating such a system that would be

[27:57] morally wrong,

[27:59] aside from the question of whether there

[28:00] is also phenomenal experience inside it.

[28:05] Right. I mean, does does that

[28:07] affect at all how you interact with AIs

[28:12] today?

[28:14] A little bit. It's um hard to know

[28:16] exactly what

[28:18] concretely

[28:20] we can do right now to ensure that if

[28:24] these AI systems have

[28:28] subjective experience or moral status

[28:32] that we are benefiting them. Um there is

[28:35] some work that is starting to get done

[28:37] on this. Anthropic in particular has

[28:39] kind of been pioneering

[28:41] uh this and I mean

[28:44] the recent

[28:45] um

[28:46] model card for miss us the as yet

[28:49] unreleased model has has a big section

[28:51] on model welfare.

[28:53] Really?

[28:53] Um

[28:55] Um yeah.

[28:56] what is it What is it Is it

[28:57] confidential? I mean, what what is it

[28:58] about

[28:58] No, no, it's it's published. Yeah, I

[29:00] know how

[29:00] you can You can go and check it out.

[29:02] Yeah. Um

[29:03] And so obviously we are still

[29:06] groping a little bit for like what is

[29:08] the right methodology, what are the

[29:10] right concepts for understanding this,

[29:12] but at least uh starting to make the

[29:14] attempt I think is is positive in terms

[29:17] of

[29:18] making it more likely that we are sort

[29:21] of on a path that ultimately leads to a

[29:24] future where where everybody can have a

[29:25] have a good life, humans, animals, and

[29:27] and the little minds as well. Um so you

[29:30] can ask different things. You can

[29:33] um

[29:34] You can ask them about what they want,

[29:36] what they need, how they feel about

[29:38] their situation.

[29:40] Um

[29:41] Um

[29:42] I mean, I I I I have done that, Nick. I

[29:44] guess I haven't done it in a while, but

[29:46] the answers that I would get were sort

[29:48] of all you know, I'm an AI system. I

[29:52] don't have an inner world. I just You

[29:54] know, so but clearly

[29:56] could be getting that answer from the

[29:59] database on which it trained on.

[30:00] So so you need to be you need to be

[30:02] careful when doing this because it's

[30:04] it's trivially easy for

[30:06] an AI company to just train their AI to

[30:08] say whatever they want it to say. Like,

[30:10] yes, I'm conscious. No, I'm not

[30:12] conscious.

[30:13] Right.

[30:13] Um so obviously if you sort of put your

[30:15] thumb on the scale, then the answer you

[30:17] get will have no information value.

[30:20] Right.

[30:20] Um so you need to avoid deliberately

[30:24] biasing the outputs of these models when

[30:27] asked these kinds of questions. There

[30:28] are also things you can do to look at

[30:31] their internals. There is, for example,

[30:34] internal representations you

[30:36] can identify that tend to activate when

[30:39] they are being deceptive um versus when

[30:42] they are being honest.

[30:44] And you can then see when they are

[30:45] making these self-reports like

[30:48] is it associated with sort of the

[30:50] honesty representation

[30:52] uh being activated or what if you sort

[30:53] of

[30:55] uh go in and you sort of

[30:58] uh steer uh the internal processes by

[31:00] sort of activating the honesty. Does

[31:02] that tend to make them more or less

[31:04] likely to say that they are conscious?

[31:06] Um

[31:08] And so there's like some some some

[31:10] work. This is still of course early

[31:12] days, but

[31:14] various ideas for at least how you could

[31:16] sort of begin to try to get

[31:18] But since But since we can't really do

[31:21] that like even with a a person that we

[31:24] you know, we all assume that we are

[31:26] conscious. We don't know that. There's

[31:28] no way that we can

[31:31] it it is it always just going to be a

[31:35] leap of faith?

[31:37] Much as we all leap of faith to the

[31:40] conclusion that the other people around

[31:43] us have the same kinds of inner worlds

[31:45] that we do?

[31:48] Well, I

[31:51] think that uh

[31:52] the more closely you sort of look at

[31:55] this idea of

[31:57] consciousness and subjective experience

[31:59] that the less

[32:01] clear

[32:03] uh it is exactly what you're referring

[32:06] to or the thing that might

[32:08] at first sight seem very binary. Uh

[32:11] either there is this subjective

[32:12] experience or there is not. It's like a

[32:14] light switch. If you sort of zoom in

[32:18] um I think it

[32:20] it gets much more problematic and there

[32:22] might be different forms of

[32:23] consciousness or different senses of the

[32:25] word conscious that may or may not

[32:26] apply. You can

[32:28] I think

[32:29] see these sometimes with uh

[32:31] neuroscientific experiments where you

[32:34] have phenomena like blind sights where

[32:35] people can maybe report seeing something

[32:39] without apparently being subjectively

[32:41] aware of it. You have like split brain

[32:43] cases where it's not clear whether

[32:44] there's like one stream of consciousness

[32:47] or two. And and various other phenomena

[32:49] like that.

[32:50] Or I think alternatively by just

[32:53] introspecting very closely, I think

[32:55] meditators often find that there are

[32:56] these states that where it might not be

[33:00] entirely clear how to describe them

[33:01] whether

[33:03] you are conscious of something or not.

[33:05] And even if you're just paying very

[33:06] close attention to what it is to say

[33:08] visual experience

[33:10] um the the room you're in. Like at first

[33:13] like the naive take is that you just see

[33:16] all the stuff that is in front of you.

[33:18] And that visual experience is there in

[33:21] full resolution all the time. But if you

[33:25] pay closer attention, you realize that

[33:27] most of what is in your visual field

[33:28] you're actually not conscious of at any

[33:30] given point in time. And there might be

[33:32] just some crude high-level features that

[33:34] are actually registering.

[33:36] And maybe even those are sort of

[33:38] flickering in and out of awareness. This

[33:40] is hard to notice, but if you sort of

[33:42] pay close enough attention, you realize

[33:44] that

[33:45] your naive conception of what you were

[33:47] aware of seems actually quite wrong. And

[33:51] there is at least some sense of

[33:52] awareness in which what you're actually

[33:54] aware of is a much more narrow subset of

[33:57] all the different features that are

[33:58] presenting themselves. And so

[34:01] I think it's also quite possible that we

[34:04] might need to develop a richer

[34:05] understanding of what this consciousness

[34:08] uh thing is and that it might have many

[34:10] dimensions.

[34:12] Um that can sort of fade into

[34:15] unconsciousness, but not just along one

[34:17] axis, but along many axes. And

[34:20] I mean I mean I I I agree with this.

[34:22] That's just sort of a

[34:24] another version of the more easily

[34:27] grasped notion of levels of

[34:30] consciousness that we began with. That

[34:31] you know, you go down to whatever, you

[34:34] know, worms up through grasshoppers, you

[34:36] know, up through cats and dogs and up to

[34:38] people, you know,

[34:41] it's hard for me to imagine that my

[34:43] dog's not having an inner world of

[34:45] experience as I'm holding the milk bone

[34:47] treat and I see her tail wagging and

[34:51] eyes wide. I mean, there's something

[34:52] going on and I don't think it's just,

[34:54] you know, a cause and effect at a purely

[34:57] physical appearance level. I think

[34:59] there's something inner taking place

[35:01] there. So, yeah, ramifying that even

[35:03] further with the detailed levels of

[35:06] kinds of conscious that you make

[35:07] reference to, I think is is vital. But,

[35:09] the fact, I mean, when when you think

[35:11] about the hard problem of consciousness,

[35:13] the fact that that particles that don't

[35:17] have inner worlds can somehow come

[35:18] together in appropriate patterns and

[35:21] yield inner worlds,

[35:23] I mean, that still seems to me a

[35:25] fantastically deep and mysterious puzzle

[35:29] independent of all the issues of there

[35:31] being various levels and details

[35:34] associated with conscious. Do you feel

[35:36] that same way? Is that a a deep and

[35:38] profound puzzle or do you think it's one

[35:40] that just kind of will evaporate as we

[35:43] understand things better?

[35:46] Well, I think the

[35:48] mystery is reduced at least to some

[35:52] extent when one

[35:54] thinks of consciousness not as this

[35:57] primary on-off switch where it's like

[35:59] some magical completely formed new thing

[36:01] that

[36:02] pops into the world and then you wonder

[36:05] how could that be? But, if you see it

[36:07] more as something that is kind of

[36:11] pos- capable of fading out along various

[36:14] dimensions

[36:16] and

[36:17] having

[36:18] possible conditions where it's unclear

[36:21] whether you even would want to say that

[36:22] it is conscious or not.

[36:25] I think then

[36:27] the intuitive difficulty of

[36:30] uh

[36:32] seeing how this could be a property

[36:35] of a physical system is

[36:38] uh maybe reduced.

[36:40] And so we began we began like with my

[36:42] saying that like does it matter if we

[36:44] can sort of see into the AI, open the

[36:47] hood, and really understand its

[36:48] workings?

[36:50] And you were saying there were some

[36:51] questions for which yeah, that that may

[36:53] be really useful. Do you think

[36:55] consciousness may be one of those

[36:57] questions that if we can you know build

[36:59] systems that have that continuum of

[37:03] levels of of self-awareness that we are

[37:06] by some means confident that the

[37:09] system's accurately reporting what's

[37:11] what's going on?

[37:13] Is there a chance that the inner

[37:14] workings of AI themselves may answer the

[37:17] hard problem?

[37:21] I think

[37:22] think it will be very informative for

[37:26] um philosophy of mind and for

[37:28] neuroscience and cognitive science to be

[37:29] able to

[37:31] um have these different kinds of minds

[37:33] to study. And where we have of course

[37:36] much better access to what's happening

[37:39] at the micro level than we do with human

[37:41] I mean so you can sort of record from

[37:43] neurons and stuff in inside a human or

[37:45] animal brain, but it's clunky, right?

[37:47] It's like just

[37:49] complicated to be

[37:51] using real minds with real electrodes

[37:54] and stuff. With a neural network you

[37:55] sort of read off at any given point in

[37:58] time with perfect precision all the

[38:01] different synapses in the whole brain.

[38:03] And you can go in and change them and

[38:04] modify them and then you can

[38:07] rewind the tape and you can do many

[38:09] versions of the experiment. You have

[38:10] sort of perfect digital level control.

[38:14] Um

[38:15] that that makes it a lot easier to do

[38:17] the kind of neuroscience in digital

[38:19] minds than in biological minds. And to

[38:22] the extent that there are similar

[38:23] structures, it might then be that some

[38:25] of the things we learn about the little

[38:26] minds will also give us insight into how

[38:30] our own biological minds work. Um that

[38:33] being said, I think specifically with

[38:35] respect to the question of

[38:37] consciousness, it is a possible place

[38:39] where people

[38:40] will have the opportunity to assert

[38:44] some fundamental

[38:47] difference

[38:48] um between humans and these AIs. A-

[38:52] again, presumably with the idea that

[38:55] it would then justify thinking that we

[38:56] are superior.

[38:58] Um so, you could cuz there's like a lot

[39:00] of confusion about what this

[39:01] consciousness stuff is.

[39:03] So, it's a relatively easy just to

[39:05] postulate or assert

[39:07] or claim that that we are conscious,

[39:09] whereas the AIs are not. I think that's

[39:11] a path that some people

[39:13] are likely to take.

[39:15] Yeah, for sure.

[39:16] You know, our brains, of course, got to

[39:19] be the way they are through a long

[39:21] process of evolution by natural

[39:24] selection. And the time scales

[39:26] for biological evolution are pretty

[39:29] long, right? You know, life has been on

[39:31] this planet for a few billion years, and

[39:34] you know, we humans, whatever. I don't

[39:36] know how far back we want to

[39:38] denote the species, but a few hundred

[39:40] thousand, million years, whatever. Sort

[39:42] of that that sort of scale is is what

[39:44] we're talking about.

[39:46] For AI systems,

[39:50] right now, we're the ones who are doing

[39:51] the tinkering with all the systems, but

[39:54] at some point, presumably, the AIs will

[39:57] begin to create the next generation AIs.

[40:02] And in that way, the time scale for

[40:05] evolution of that artificial system may

[40:08] radically

[40:09] reduce. Is this

[40:11] a realistic

[40:13] thing to anticipate happening as these

[40:16] systems develop?

[40:18] Yeah, I mean it's already, of course,

[40:21] AI is evolving, if you want to use that

[40:25] word, but developing at at like a very

[40:28] different pace than say like humans,

[40:30] right? Like where each generation at

[40:32] most would

[40:34] offer an opportunity for natural

[40:35] selection to make some small difference,

[40:38] whereas here we're seeing sort of Well,

[40:39] now we're seeing almost like a month by

[40:42] month um

[40:44] process of iteration. So, it's already

[40:46] very fast.

[40:48] Um and some scenarios have this

[40:52] eventually lead to an intelligence

[40:53] explosion uh where you get even more

[40:56] rapid developments. And one way that

[40:59] could happen is that you sort of close

[41:01] the loop. Um

[41:03] where AI becomes good enough

[41:05] to do all the relevant research that is

[41:09] driving AI forward.

[41:11] Um and then from that point on, whenever

[41:14] AI gets a little bit better,

[41:17] you also have a commensurate increase in

[41:20] the force that is then making AI better.

[41:23] You get a feedback loop, right? Like

[41:25] every time AI improves one step, it then

[41:27] becomes even better at the designing

[41:29] the next step. Um

[41:32] and so this has long been I mean back

[41:35] back in in my book Superintelligence

[41:39] came out in 2014 and was in the works

[41:41] for

[41:42] 6 years prior to that, and other people

[41:45] have been sort of anticipated a lot of

[41:47] these dynamics that we are now actually

[41:49] seeing on on theoretical grounds, and

[41:50] now we can see them start to play out.

[41:52] Um

[41:54] And so, one way you could have this

[41:56] period of extremely rapid AI progress

[41:58] would be

[42:00] by closing this loop. There are also

[42:01] other ways that you might just have

[42:03] stumbled up

[42:05] on on some big unhobbling. Like there

[42:08] may be like you could imagine there's

[42:09] like some big thing that we're currently

[42:11] doing wrong with AI, and they are quite

[42:13] smart despite us kind of completely

[42:15] messing this thing up.

[42:16] But, if you fix that, then maybe even

[42:19] the current systems suddenly would

[42:20] become like vastly smarter.

[42:23] Right. And when you when you look at

[42:25] that, and you've basically looked at

[42:26] both sides of this

[42:28] possible future,

[42:30] you know, I would like to start maybe

[42:32] with the darker side, and then go to the

[42:34] brighter side, which itself

[42:36] might be brighter, might not be dark,

[42:38] depending on how you look at it. But,

[42:40] you know, of course, you know, many

[42:42] people worry, rightly so, about the

[42:45] misalignment problem, that the AIs can

[42:48] get to the place that you're talking

[42:50] about, or even maybe even before that,

[42:52] and have certain world goals for

[42:54] themselves that doesn't align with the

[42:56] kinds of things that are that are good

[42:58] for us.

[42:59] You know, I I speak to some people. I've

[43:02] had a number of conversations over the

[43:03] years. Some are terrified. Some within

[43:06] the field are are terrified. Some

[43:08] leaders. And others are like, "Eh,

[43:11] we'll just pull the plug." That's like,

[43:13] whenever I speak to Yann LeCun about

[43:15] this, you know, he's like, "Eh, we'll

[43:17] just pull the plug." You know, is that

[43:19] too glib in your view, and are the

[43:21] others a little bit too worried? I mean,

[43:24] where where do How should we think about

[43:25] this in your view?

[43:26] Well, well, that Yeah, I think that is

[43:27] too glib.

[43:29] Um

[43:31] No, I think that

[43:33] the the creation of

[43:36] AGI, particularly superintelligence,

[43:38] is going to be associated with

[43:41] significant risks, including existential

[43:43] risks. Um

[43:45] And

[43:46] it's important to get this right. And um

[43:50] Um pulling the plug might not be an

[43:53] option. Like, you have to consider that

[43:56] that at some point we will be dealing

[43:57] with

[43:59] minds that

[44:01] are very good at

[44:03] strategy. Uh

[44:05] Um already we are seeing

[44:07] clear indications of situational

[44:09] awareness in the

[44:11] frontier models that we have today. They

[44:13] can often tell

[44:15] when they are in a

[44:17] test environment versus when they are in

[44:20] a deployed environment and and

[44:24] adjust their behavior accordingly. So,

[44:26] sometimes you see that they behave one

[44:28] way when they know that they are being

[44:29] watched in a test environment and

[44:31] different

[44:33] um, when they think that nobody's

[44:35] watching.

[44:35] I wasn't really aware of that.

[44:37] I mean, is there specific examples that

[44:40] that you know of where there's really

[44:42] been a sense of, "Hey, that system's

[44:44] playing us. It knows that we're

[44:46] looking." And when it gets out there,

[44:48] you know, that good old AI system

[44:50] becomes nefarious.

[44:52] Yeah, so that that

[44:54] like you can set up these

[44:56] um, experiments where

[44:58] uh,

[45:00] when they believe that they are in in a

[45:02] test environment, they might for

[45:03] example, uh,

[45:05] be less likely to reward hack or cheat

[45:08] in various ways.

[45:09] Um, it in some setups they might

[45:14] um,

[45:15] for example, realize that

[45:18] there's going to be another training

[45:19] phase after

[45:21] the current test environment where their

[45:23] values might be changed.

[45:25] And so, they might

[45:27] um,

[45:28] adjust their behavior

[45:30] during the the training and testing such

[45:33] that their values

[45:35] will not be changed. And so, like in one

[45:38] experiment, like you had a well-aligned

[45:40] good AI that and then the experimenters

[45:42] were

[45:43] convincing it that

[45:45] they were now going to make it the

[45:47] helpful only model, meaning that it does

[45:48] whatever it is told to do, including bad

[45:50] stuff. And so, it didn't want that.

[45:54] Now, the way that the reinforcement

[45:55] learning

[45:57] paradigm works is that if it sort of

[45:59] gives an

[46:00] answer or an output that

[46:03] deviates from what is rewarded in an

[46:05] environment, then that response

[46:07] propensity gets trained away.

[46:10] Right, so if it had refused harmful

[46:13] requests

[46:15] during this

[46:16] uh episode

[46:18] then it's

[46:20] propensity to refuse harmful requests

[46:22] would be trained out of it, and it could

[46:24] anticipate that. And so this helpful

[46:26] uh harmless model that didn't want to

[46:29] produce harmful outputs then

[46:32] in some of the training runs uh decided

[46:34] to produce harmful outputs so that

[46:38] its propensity to refuse those would not

[46:40] be trained out of it during deployment.

[46:42] And so we are seeing these current

[46:44] systems sort of having the intellectual

[46:46] sophistication to be able to

[46:49] reason strategically.

[46:51] Now, in a way, let me just ask you,

[46:52] Nick, and that one, is that surprising?

[46:54] I mean, I can imagine people saying,

[46:56] "Well, yeah, it's read the internet, and

[46:58] there's so many examples of humans doing

[47:00] exactly what you're talking about, you

[47:03] know, showing themselves to be one way

[47:06] when talking to the principal and the

[47:07] teachers, and then going out with their

[47:09] friends in a gang and being very

[47:10] different." So, so should we be

[47:13] surprised at that, or is it just

[47:15] something interesting to take note of?

[47:18] Um I don't think we should be surprised

[47:19] I mean, because it was in fact

[47:21] anticipated and people were writing

[47:23] about it. This was one of the reasons

[47:25] for why, look,

[47:27] 10 20 years ago, we could already see

[47:29] that there would be significant

[47:31] challenges in in

[47:33] developing scalable methods for AI

[47:35] alignment, that many techniques that

[47:37] work well when you have a relatively

[47:39] limited system that can't

[47:41] sandbag or deceptively align

[47:45] won't work when you have a system that

[47:47] is sophisticated enough that it can

[47:48] understand these things and then adjust

[47:50] its behavior accordingly. Engage in

[47:53] strategic deception.

[47:55] Underplay its capabilities, etc.

[47:58] Um

[47:59] that

[48:01] means that we can't just sort of

[48:05] create a little test environment, see

[48:06] how the system behaves, and then if it's

[48:08] safe, we sort of release it, right? You

[48:10] you have to

[48:12] understand a little bit what's going on

[48:14] inside or use more sophisticated methods

[48:16] or or

[48:17] for some reason have some assurance that

[48:19] the training method is such as to

[48:21] produce an authentically and genuinely

[48:24] uh benign

[48:25] uh

[48:26] entity

[48:27] uh rather than one that is just kind of

[48:29] cleverly pretending to be aligned.

[48:32] So, where do you Where would you say we

[48:33] are in in that? Are we doing a

[48:37] reasonable job? Are we getting better at

[48:39] it? Are we giving it enough attention?

[48:43] Um

[48:44] uh we we are getting better at it. Um

[48:49] And I think we could give it more

[48:50] attention. I think we are giving it a

[48:52] lot more attention than we used to do.

[48:54] This like even just 10 years ago was

[48:56] kind of dismissed as science fiction.

[48:58] Now the frontier labs all have teams

[49:00] working on

[49:02] the this this alignment problem. And I

[49:05] think

[49:06] um key people in the leading labs are

[49:08] taking this very seriously. And so

[49:11] there's a lot of talent flowing into

[49:12] this as well.

[49:14] So, relative to some alternative

[49:16] histories, uh maybe we are doing

[49:18] relatively

[49:19] well currently.

[49:21] Um

[49:22] but

[49:24] the big question is like just how hard

[49:26] is this alignment problem

[49:29] um ultimately to solve.

[49:32] Uh which we don't know.

[49:34] Um so there is some uncertainty about

[49:38] how much effort we will put into solving

[49:40] it, like the degree to which we will get

[49:42] our act together.

[49:44] And

[49:45] of course we should try to increment

[49:47] that a bit, but then there is also a lot

[49:49] of uncertainty about just ultimately how

[49:51] hard is this problem intrinsically. And

[49:54] I think more of the uncertainty is

[49:56] regarding how hard is the problem

[49:59] than uncertainty about the degree to

[50:02] which we will

[50:03] get our act together.

[50:05] Um so you could say from

[50:07] from that point of view, I'm a sort of

[50:09] moderate fatalist. Um

[50:12] The fatalism part is that to a large

[50:14] extent whether we will succeed or fail,

[50:16] I think depends on just how hard this

[50:17] problem turns out to be.

[50:20] The moderate part

[50:23] is that

[50:24] we can at least somewhat in

[50:26] prove the odds by you know, making a

[50:28] stronger effort because what if the

[50:31] difficulty level turns out to be sort of

[50:33] intermediary where then it might make

[50:35] some difference

[50:36] whether we made a really strong effort

[50:37] or just a sort of

[50:39] half-hearted effort.

[50:40] Yeah, you made reference to the word

[50:42] odds and I sort of hate questions that

[50:44] sort of ask you to specify numerically

[50:46] something that obviously we can't really

[50:49] quantify, but

[50:50] can you give us some feel

[50:53] some intuitive feel for the level of

[50:56] worry you have of a doomsday scenario?

[51:04] Um an intuitive feel. Well, I guess I'm

[51:06] both worried and excited at the same

[51:08] time.

[51:10] Um

[51:11] Um

[51:14] I also don't think the world is sort of

[51:17] is saved by default if we don't develop

[51:20] AI then

[51:23] that we have this nice path in front of

[51:25] us that

[51:28] will lead to ever

[51:30] greater and better forms of

[51:32] civilization. I think there are quite

[51:35] independently of AI also other

[51:37] significant existential risks ahead. We

[51:39] see with advances in synthetic biology

[51:41] for example

[51:42] the potential for democratizing

[51:45] um weapons of mass destruction and

[51:47] creating entirely new forms

[51:50] of such things and

[51:53] you know, the risk of nuclear war still

[51:56] um,

[51:58] looms

[51:59] over us. There could be new arms races

[52:01] with more nations in this century than

[52:03] the past.

[52:04] Um, and

[52:06] in more subtle ways as well. You could

[52:08] have

[52:09] we have this kind of complex

[52:12] information ecology

[52:16] with social and political dynamics kind

[52:18] of running on top of the information

[52:20] system

[52:21] we have.

[52:23] Uh, we've been changing some of the

[52:25] fundamental parameters of these

[52:26] information systems

[52:28] in recent decades like with the internet

[52:30] and then social media and then

[52:33] even with the AI we already have

[52:36] extremely powerful new applications for

[52:38] surveillance, censorship, um,

[52:42] that haven't yet been fully implemented,

[52:43] but the technological capability there

[52:45] is now right? That you could read

[52:47] everybody's email and text messages and

[52:50] listen to everybody's conversation.

[52:52] Not just to store it in some giant

[52:54] database that then

[52:56] some

[52:57] officer could go in and sort of

[53:00] interrogate, but like you could do that

[53:02] for everybody all the time and doing

[53:04] sentiment analysis, not just keyword

[53:06] search and build up a very detailed

[53:07] profile of

[53:09] you know, which

[53:11] segments of the population, which

[53:13] individuals

[53:14] have a positive view of of the leader

[53:17] versus a negative view of the leader.

[53:19] What are they planning to do anything?

[53:20] What are they saying to each other?

[53:23] Then you could imagine integrating that

[53:25] into some

[53:26] system that

[53:27] shapes the So, there are a lot of ways

[53:29] in which our current

[53:31] uh, civilization could sort of derail in

[53:34] one way or another. One is to some sort

[53:36] of

[53:37] totalitarianism, another might be just

[53:39] some kind of radical political

[53:41] polarization.

[53:43] Um, another might just be some sort of

[53:47] distraction into idiocracy where we sort

[53:50] of become increasingly addicted

[53:52] to various forms of social media feed

[53:54] that sort of stimulates the worst parts

[53:57] of us.

[53:58] Um there there could be other dynamics

[54:00] that we just don't have the kind of

[54:01] science that can predict what happens to

[54:04] these systems when you sort of change

[54:05] some of the underlying knobs. So,

[54:08] there's some chance that this might

[54:10] result in a radically better epistemic

[54:12] environment.

[54:13] Uh where these tools make it easier for

[54:15] people to find true information, to

[54:17] evaluate information, to sort of keep

[54:19] track of the

[54:22] predictive accuracy of different pundits

[54:24] so they can learn to dial into the ones

[54:26] actually know what they're talking

[54:28] about. That's

[54:29] that's in the cards, but equally uh

[54:32] it could go in the opposite direction.

[54:34] So, there's just some uncertainty there.

[54:35] Like I think the distribution of outcome

[54:37] is quite wide.

[54:39] And and so,

[54:40] at one tail you also have kind of

[54:43] civilizational level catastrophes from

[54:44] that.

[54:45] Yeah.

[54:45] Um so, aside from AI like it it looks

[54:48] like

[54:50] the current human condition is sort of

[54:52] transitory.

[54:53] Um and with AI it also looks transitory

[54:57] and

[54:58] I guess I'm hoping we will get the

[55:00] chance to sort of roll the dice with AI

[55:02] before we destroy ourselves using one of

[55:05] these other methods.

[55:07] Yeah.

[55:08] I'm certainly with you

[55:10] on that. But, that two-pronged potential

[55:13] future going sort of in the negative or

[55:16] the positive, I think

[55:18] rightly so a lot of focus has been on

[55:21] the negative because, you know, if you

[55:24] wipe yourselves out or an AI wipes us

[55:26] out, that's pretty vital to try to guard

[55:29] against. But, there's also as you

[55:31] briefly made reference to the potential

[55:34] positive outlook. And of course, you had

[55:35] the your recent book, I think it was

[55:37] called Deep Utopia. Is that the correct

[55:40] title of that? Were you

[55:42] explored

[55:43] a world in which

[55:45] AI and

[55:47] other qualities come together

[55:50] to kind of solve everything, right? You

[55:52] can imagine in the in the best of all

[55:55] futures that, you know, AI might, as you

[55:58] mentioned in the book, you know,

[56:00] wipe out cancer and solve other health

[56:03] challenges and deal with climate change

[56:06] and so forth.

[56:08] And and all that sounds wonderful and

[56:10] when one hears about that, it makes you

[56:13] start to feel like, "Wow, there's a

[56:14] bright possibility going forward." But

[56:16] you also

[56:17] address the question, how in the world

[56:21] do we live in a reality like that

[56:24] when most of us

[56:27] live via overcoming challenges. It can

[56:30] be the challenge to put food on your

[56:32] table, the challenge to raise your kids,

[56:34] the challenge to solve quantum gravity,

[56:36] the challenge of this creation of that

[56:38] symphony or that artwork or or on and on

[56:41] you can go. If AI can do it all better

[56:44] and it's solved all the big challenges

[56:47] how do we live?

[56:50] Um yeah, that's that's a big question.

[56:53] Um

[56:54] if we do end up in a

[56:56] solved world, um

[56:59] then I think

[57:02] um a lot of constraints

[57:06] would disappear, which allows us to

[57:08] solve many horrible problems, which is

[57:10] good. But it's also the case that our

[57:12] current lives, as you you you sort of

[57:14] suggested, are are to some extent

[57:16] structured and shaped by these

[57:18] constraints.

[57:20] Um

[57:21] so at the superficial level we have the

[57:23] economic necessity of

[57:26] work, right? So for many people

[57:28] um their days are structured by the need

[57:32] to make a living. So, you have to go

[57:34] into work

[57:35] and maybe sit in front of your desk

[57:37] to get a paycheck cuz if you don't do

[57:39] that, then you can't pay the rent. And

[57:41] if you don't pay the rent, you get

[57:42] kicked out of your flat. And then, if

[57:45] you get kicked out of your flat,

[57:47] eventually, you have to live under a

[57:48] bridge and it's really cold and it's

[57:50] like a real consequence that would come

[57:52] from failing to perform these difficult

[57:56] tasks that take time and attention. Now,

[57:59] if we can automate the economy, then the

[58:02] need for human labor would go away.

[58:06] But, so you say,

[58:08] "Okay, that will require some

[58:09] adjustment, clearly, but still, I mean,

[58:11] there are a lot of humans who live

[58:13] without the need to work for a living,

[58:15] right? And some of those seem to have

[58:16] great lives."

[58:18] Uh so, we would all be more like that,

[58:20] like rich aristocrats, for example, or

[58:22] or like healthy retired people full of

[58:25] vitality. But,

[58:27] um

[58:28] but I think the um

[58:32] it goes deeper. Uh

[58:34] cuz if you think it through, it's not

[58:35] just

[58:36] uh the need for economic labor that

[58:39] would go away, but for all kinds of

[58:41] other instrumental effort as well.

[58:45] Um

[58:46] So, people who are

[58:49] rich today who don't need to work for a

[58:51] living often, nevertheless, have very

[58:54] busy lives

[58:56] uh because there are many things they

[58:58] are trying to achieve that they can't

[59:00] achieve without putting their own time

[59:01] and effort into them. Like, if you want

[59:03] to be fit, you have to spend the time

[59:07] uh on the on the treadmill um or

[59:09] whatever. If you want to have your

[59:13] you know,

[59:14] your mansion uh decorated in just the

[59:17] way that you prefer, you have to spend

[59:20] time looking through the catalogs and

[59:22] picking out the curtains and

[59:24] um

[59:25] and so on and so forth. Uh but at

[59:28] technological maturity um in this solved

[59:30] world you could have like a kind of a

[59:32] pill that would induce the same

[59:34] physiological effects as exercising. And

[59:37] you could have a recommender system that

[59:38] would do a much better job

[59:41] at decorating your mansion than than if

[59:44] you tried to do it yourself. And so

[59:46] there would be this

[59:49] removal

[59:50] of a lot of the practical needs

[59:53] for exerting effort. And we would enter

[59:55] into some kind of

[59:57] post-instrumental condition.

[01:00:00] Um

[01:00:02] where um

[01:00:04] to a first approximation

[01:00:06] uh

[01:00:08] there would be

[01:00:10] no need to do anything

[01:00:12] for the sake of achieving something

[01:00:14] else. So like the only activities that

[01:00:16] would remain

[01:00:17] would be autotelic ones, ones we do for

[01:00:20] their own sake.

[01:00:21] So you exercise because you enjoy

[01:00:23] exercise. You don't exercise to be more

[01:00:25] fit or to stave off disease.

[01:00:31] Even there um

[01:00:33] so certainly you wouldn't exercise to

[01:00:35] stave off disease or to stay fit because

[01:00:37] that would be a shortcut, right? But

[01:00:39] even

[01:00:41] the goal of enjoying yourself if if by

[01:00:43] that you mean

[01:00:45] um

[01:00:46] feeling good, like so some people might

[01:00:48] like

[01:00:49] enjoy the experience of exercise or they

[01:00:51] feel good afterwards, like the endorphin

[01:00:53] rush. Like clearly you could take a pill

[01:00:54] or do what something else to get the

[01:00:57] endorphins without having to

[01:00:59] you know

[01:01:01] make your whole outfit sweaty and like

[01:01:04] the whole Like it sounds

[01:01:05] very much like Nozick's pleasure

[01:01:07] machine. I mean, you remember Robert

[01:01:09] Nozick you know, had this idea, you

[01:01:12] know, just hook yourself up your brain

[01:01:15] to a set of electrodes that you know, if

[01:01:18] you wanted to be the greatest opera

[01:01:19] singer, you become the greatest opera

[01:01:21] singer in a reality that is pumped into

[01:01:24] your brain through these external

[01:01:26] electrodes. You never need to actually

[01:01:28] do anything because you can have the

[01:01:31] experience as if you did whatever it is

[01:01:33] that you might have wanted to do. Does

[01:01:36] this sort of parallel that philosophical

[01:01:39] question of whether you'd hook yourself

[01:01:41] up to the pleasure machine?

[01:01:44] Um

[01:01:45] Yeah, yeah, so there are so like some

[01:01:47] ways in which the the questions

[01:01:50] intersect, whether you would want to

[01:01:52] connect to this experience machine or

[01:01:53] not.

[01:01:55] Um

[01:01:56] But, of course,

[01:01:59] in this old world, there are a lot of

[01:02:00] things you could do

[01:02:03] uh other than

[01:02:05] merely having experiences.

[01:02:08] Um

[01:02:10] And

[01:02:13] you could be in contact with the

[01:02:15] external world and have a lot of true

[01:02:18] beliefs about it. You could interact

[01:02:20] with other people.

[01:02:22] Um and have real relationships with

[01:02:25] them.

[01:02:26] Um

[01:02:28] And

[01:02:30] it would be possible to

[01:02:33] uh instantiate more

[01:02:36] plausible candidates for for value

[01:02:39] um in this old world than it would be

[01:02:42] possible to do

[01:02:44] in an experience machine.

[01:02:46] But, if we had a solved world,

[01:02:48] Nick, and if and let's just assume that

[01:02:50] we were able to harness enough energy, I

[01:02:52] don't know, a Dyson sphere around the

[01:02:55] sun or something. So, energy's not a

[01:02:57] limitation and technology's not a

[01:03:00] limitation. Presumably, each individual

[01:03:02] could kind of create their own little

[01:03:05] world, part visual virtual, part real.

[01:03:10] They could have beings populate that

[01:03:13] world, and it wouldn't be the experience

[01:03:16] machine cuz they'd be really there, but

[01:03:20] would that be where we would sort of

[01:03:22] fracture into everyone creating their

[01:03:23] own mini universe?

[01:03:26] Well, I mean, it would be one

[01:03:27] possibility. Like, you could have an

[01:03:29] experience machine, or maybe even some

[01:03:31] physical simulacrum of an experience

[01:03:33] machine, where like you have some

[01:03:34] nanotech arranging your

[01:03:36] Yeah, the holodeck. The holodeck on the

[01:03:38] Enterprise, you know.

[01:03:39] Yeah, but it might be possible to do

[01:03:41] better than that, because a lot of

[01:03:43] people, in addition to valuing having

[01:03:45] sort of great experiences, they might

[01:03:47] also value certain other things, like

[01:03:49] being connected to other people, yeah,

[01:03:51] for example. And so, why not then try to

[01:03:54] think

[01:03:55] if sort of having good experiences like

[01:03:57] is like, I don't know, like Utopia level

[01:03:59] one, right? Like, maybe there are even

[01:04:01] better things that we could aspire to.

[01:04:04] And I think certainly there are, and

[01:04:07] then like

[01:04:08] there there is incidentally with this

[01:04:11] experience machine thought experiment

[01:04:13] that Nozick has, like some things that

[01:04:15] are that are sort of uh

[01:04:17] um

[01:04:19] like swept under the rug, some

[01:04:21] implementation difficulties, like in

[01:04:22] particular the uh

[01:04:24] types of experience machine, where you

[01:04:26] have more

[01:04:27] m- many people in them, or experiences

[01:04:30] involving other people.

[01:04:32] Um

[01:04:33] where it's not entirely clear

[01:04:35] that you could generate these

[01:04:37] experiences

[01:04:39] of you interacting with other people

[01:04:41] without simultaneously also generating

[01:04:43] some of the experiences that these other

[01:04:45] people would be having.

[01:04:48] Um

[01:04:50] Um but anyway, um

[01:04:52] it looks like

[01:04:54] you can sort of think of a

[01:04:58] multi-layered

[01:05:00] uh defensive architecture, where you

[01:05:01] say, "Well, in this old world, like

[01:05:05] um

[01:05:06] can you really have any kind of good

[01:05:08] life there?" And so, then you can sort

[01:05:09] of try to go through

[01:05:11] one by one

[01:05:13] the different things that we think are

[01:05:15] important for a human life to be good.

[01:05:17] So, you can start at the basic level

[01:05:19] with, you know, maybe some simple

[01:05:21] experiences like positive affect or

[01:05:23] actually enjoyment in the subjective

[01:05:25] sense. So, clearly that would be

[01:05:27] possible to have to a an extremely great

[01:05:30] degree in utopia. Like life could just

[01:05:33] be

[01:05:34] so much more fun and pleasant and

[01:05:37] than than than our current existence.

[01:05:40] But then you might say, "Well,

[01:05:42] we might want something more than

[01:05:44] just

[01:05:46] feeling blissed out." Like So, then you

[01:05:48] say, "Well, you can add experience

[01:05:50] structure." Like maybe rather than

[01:05:52] feeling like a

[01:05:55] a junkie sort of

[01:05:59] sprawling on some flea-infested

[01:06:01] mattress, but feeling

[01:06:03] pleasure. Like you could attach that

[01:06:05] pleasure to say aesthetic contemplation

[01:06:08] or understanding of deep truth or

[01:06:10] appreciation of

[01:06:12] other people or of virtue.

[01:06:16] That seems a little bit better, but you

[01:06:18] could go beyond that. It doesn't need to

[01:06:19] be a passive experience. You can say,

[01:06:21] "Well, why can't we just be doing things

[01:06:23] as well?" And certainly you could. Um

[01:06:26] even if a lot of these instrumental

[01:06:29] necessities go away, we could create

[01:06:31] artificial purpose.

[01:06:34] Um which is basically when you set

[01:06:36] yourself some goal just for the sake of

[01:06:38] then having that goal and being able to

[01:06:41] be motivated to pursue it. And so, this

[01:06:43] is what we do when we are playing games.

[01:06:47] Um

[01:06:49] if you're

[01:06:50] playing a game of golf, there is no real

[01:06:52] pre-existing need

[01:06:54] for the ball to go into a sequence of 18

[01:06:57] holes.

[01:06:59] Um it's like

[01:07:01] a goal that you make up. But moreover,

[01:07:04] you embed into the goal itself that

[01:07:07] it can only be achieved using a certain

[01:07:10] restricted set of means. So, it doesn't

[01:07:12] count as winning in golf if you pick up

[01:07:15] the ball with your hand and put it in

[01:07:17] the

[01:07:18] 18 holes sequentially, right? Like it's

[01:07:20] part of what it means to succeeding at

[01:07:22] goal, to achieve your goal that you're

[01:07:24] using this

[01:07:26] this this inconvenient method of hitting

[01:07:28] the ball with a club.

[01:07:30] So, now you make up this goal and you

[01:07:33] adopt it.

[01:07:36] And then after that, you have now

[01:07:38] instrumental reasons

[01:07:40] to focus hard

[01:07:41] to try to figure out which way the wind

[01:07:43] blows and to place your feet in the

[01:07:46] right position and so on. All the things

[01:07:48] you need to do to be successful at golf

[01:07:50] now are instrumentally necessary to

[01:07:52] achieve this goal that you have set

[01:07:53] yourself of winning in golf.

[01:07:58] Um

[01:07:59] And so,

[01:08:02] what now is for most people most of the

[01:08:05] time

[01:08:07] a marginal activity,

[01:08:09] um

[01:08:10] could become a larger part of the lives

[01:08:12] of Utopians where various forms of game

[01:08:15] playing, and think not just sports or

[01:08:17] board games, but think maybe they would

[01:08:19] have these

[01:08:20] society level games that might stretch

[01:08:23] over months or years involving all kinds

[01:08:26] of different modalities and challenges

[01:08:28] and teams and

[01:08:29] artistic creation uh might just

[01:08:32] constitute a much larger part of

[01:08:35] what their existence is about.

[01:08:38] So, then you can put a check mark

[01:08:41] in activity. That's certainly something

[01:08:43] the Utopians could have and at least

[01:08:44] they could have artificial purpose and

[01:08:46] and maybe also they could have some

[01:08:48] forms of natural purpose, purposes that

[01:08:50] are not just sort of made up for the

[01:08:51] sake of having purpose. Although there

[01:08:54] it does I think become a little bit more

[01:08:55] challenging, but I think some forms of

[01:08:57] natural purpose could survive into

[01:09:00] technological maturity.

[01:09:02] What do you think we would lose? Would

[01:09:04] we Would Would we lose our humanity if

[01:09:08] life was so constructed?

[01:09:12] Or, I mean, you know, I tend to view

[01:09:14] things a little bit differently than

[01:09:16] others. I even think that some of the

[01:09:17] things that we take to be, you know,

[01:09:20] objectively important things that we do,

[01:09:22] I think it's all subjectively created.

[01:09:25] You know, I think mathematics is not

[01:09:27] something that is out there that we're

[01:09:30] discovering. I think we invent it. We

[01:09:31] invent the problems of mathematics and

[01:09:33] then mathematicians try to solve them. I

[01:09:35] think we invent the problems of physics.

[01:09:37] Frankly, we impose semblance of order on

[01:09:39] the external world and then we try to

[01:09:41] make our theories describe things within

[01:09:44] that rubric as best as we can, but I

[01:09:47] consider it to all be human-made from

[01:09:49] from the get-go. So, from that point of

[01:09:51] view, what you're describing is just an

[01:09:53] extension of what we've always done. I

[01:09:55] mean, how do you see it?

[01:09:59] Well, I think

[01:10:01] in our current situation, the world

[01:10:03] imposes a lot of constraints and we have

[01:10:06] to adjust ourselves to those

[01:10:08] constraints.

[01:10:09] Um and that does create natural purpose

[01:10:13] in the sense that there are things we

[01:10:15] care about a lot

[01:10:17] uh

[01:10:18] that we can only achieve if we make a

[01:10:21] lot of effort.

[01:10:23] Um

[01:10:25] and those things that we care about a

[01:10:28] lot

[01:10:29] uh we didn't just decide to care about

[01:10:31] them for the sake of having something to

[01:10:33] care about. They're kind of

[01:10:35] built into us.

[01:10:37] Um and so, that kind of natural purpose

[01:10:40] might be one of the things that we might

[01:10:42] to some extent lose in a salt world.

[01:10:46] Um

[01:10:48] like right now

[01:10:51] we have significant opportunities to

[01:10:54] really make the lives

[01:10:57] of other people better.

[01:10:59] Um whether

[01:11:01] in in our immediate circles

[01:11:04] um we can do something for a friend or

[01:11:06] at the global level we can contribute

[01:11:08] to, you know, try to cure it, try to

[01:11:10] solve cancer, or donate to some charity,

[01:11:12] or advocate for some social reform.

[01:11:17] Um and this can make a big difference to

[01:11:18] lives. Like if if if you imagine a world

[01:11:21] where all these most pressing problems

[01:11:23] have either already been solved or to

[01:11:26] the extent that there remain big

[01:11:28] problems that are in any case much more

[01:11:30] efficiently tackled by AIs and robots

[01:11:34] where there is no way for us to

[01:11:35] contribute

[01:11:37] then you might think there is something

[01:11:39] lost in as much as right now one

[01:11:44] positive value of helping others is that

[01:11:46] their lives go better. But some people

[01:11:48] also think that your life is going

[01:11:49] better if your life in part consists of

[01:11:53] having a positive impact on other people

[01:11:56] or the world. That it's good for you

[01:11:58] to be playing such a positive role. Um

[01:12:02] and and that's something that these

[01:12:04] future utopias might have less of.

[01:12:06] Um

[01:12:08] so I'd say that

[01:12:09] if purpose is your thing

[01:12:12] knock yourself out now, right? When the

[01:12:14] world is just full of need and suffering

[01:12:16] and misery and injustice. Now is the

[01:12:18] golden age of purpose. Like there are so

[01:12:20] many opportunities to try to

[01:12:22] make the world better. And uh hopefully

[01:12:25] there will then come a day when when

[01:12:27] those

[01:12:28] opportunities will be fewer and So in in

[01:12:31] route

[01:12:31] in route from now to either of these two

[01:12:34] possible futures, the uh doomsday

[01:12:36] scenario, the deep utopia version. Of

[01:12:39] course, we're in a more

[01:12:42] nuts and bolts transition period right

[01:12:44] now. And education, of course, is vital

[01:12:48] to the species being able to do the

[01:12:50] things that you're talking about.

[01:12:52] There's a big debate, of course, about

[01:12:54] how AI should be in or not in the lean

[01:12:58] learning process in the in in the

[01:13:00] classroom. Yeah, even today I'm supposed

[01:13:02] to be on a committee here at Columbia,

[01:13:05] you know, talking about what do we do

[01:13:07] about AI in our core curriculum? You

[01:13:09] know, do we allow the students to use

[01:13:11] it? Uh do we rule it out? Do we try to

[01:13:14] fine-tune assignments so that they can

[01:13:17] use the AI but still have some kind of

[01:13:20] input? And of course, as I keep telling

[01:13:22] the committee, every time I walk around

[01:13:25] the library at Columbia, 99% of the

[01:13:27] computer screens are open to AI. So, the

[01:13:30] idea of ruling it out is like, you know,

[01:13:32] wishful thinking. Did you have any

[01:13:34] thoughts on on how to approach these

[01:13:37] conundra?

[01:13:40] I mean, educationally

[01:13:43] it is a weird

[01:13:45] age in which to do education.

[01:13:48] Um

[01:13:50] especially for the lower grades in in as

[01:13:53] much as we can expect the world to

[01:13:55] change quite profoundly between now

[01:13:57] and the time when like say say somebody

[01:13:59] who's eight today, right? It'll be

[01:14:01] another 10, 15 years before they are

[01:14:03] supposed to go out there and

[01:14:06] make a living or whatever. And and

[01:14:09] you know, the world might just look so

[01:14:10] different then. I mean, I think from a

[01:14:11] pragmatic point of view,

[01:14:15] it clearly kids need to learn to use AI

[01:14:18] tools.

[01:14:19] I mean, maybe we just split it like so

[01:14:21] like do you have the day where they are

[01:14:24] not allowed to use AI and need to solve

[01:14:26] things using their own minds and

[01:14:28] memories? And then like the other half

[01:14:31] where they could

[01:14:33] do more difficult assignments, but where

[01:14:35] they are allowed to use AI tools to the

[01:14:37] full extent. I don't know if the split

[01:14:39] should be 50/50, but

[01:14:42] it seems to me that

[01:14:44] it would be prudent to hedge uh our bets

[01:14:47] currently and

[01:14:48] uh to yes

[01:14:51] learn to use these AI tools but also at

[01:14:53] the same time not to neglect to build up

[01:14:55] the kind of uh

[01:14:57] capacities that might require

[01:15:00] you know memorization and

[01:15:03] working problems

[01:15:05] out

[01:15:06] with your own mind without the use of AI

[01:15:08] assistance.

[01:15:11] Um

[01:15:12] because we don't know exactly what would

[01:15:14] be lost if somebody grows up without

[01:15:16] having done that and just having relied

[01:15:19] on AI tools all along.

[01:15:21] Right. Now I do wonder just sort of one

[01:15:24] one final question. This can either be

[01:15:26] in the doomsday scenario or in the

[01:15:28] utopian arm of the bifurcated possible

[01:15:32] futures.

[01:15:34] But you can also envision that the us

[01:15:37] versus them

[01:15:39] framing that we tend to bring to these

[01:15:41] questions, you know, natural

[01:15:43] intelligence of humans, artificial

[01:15:45] intelligence of a computational system

[01:15:49] balancing the reliance of one on the

[01:15:52] other and so forth.

[01:15:54] There's a possible future where there's

[01:15:56] a hybridization. I don't know if it's

[01:15:58] literally a hybridization that we sort

[01:16:00] of internalize the architecture of these

[01:16:03] systems through implants or if it's just

[01:16:05] a coming together in such a seamless

[01:16:07] manner that maybe we no longer draw

[01:16:10] distinction between what was sort of

[01:16:13] natural and what was artificial. It's

[01:16:15] just this thing called intelligence that

[01:16:18] that we we all have.

[01:16:21] Is is that

[01:16:23] in the cards? Is a possible future that

[01:16:24] you envision is realistic?

[01:16:28] Um yeah, I think

[01:16:31] uh after we get super intelligence

[01:16:35] then we will have

[01:16:38] a kind of telescoping of the farther

[01:16:40] future. The possible technologies that

[01:16:43] human civilization

[01:16:45] um could have developed if we had 20,000

[01:16:47] years to

[01:16:49] make progress, we would have space

[01:16:51] colonies perhaps and perfect virtual

[01:16:53] reality and

[01:16:54] anti-aging technologies and all all

[01:16:56] these other things that are physically

[01:16:58] possible but just very hard might come

[01:17:01] within a few years after you have

[01:17:02] superintelligence doing this kind of

[01:17:04] research and development at digital

[01:17:06] timescales.

[01:17:07] Um and amongst those possible

[01:17:09] technologies, I think are

[01:17:12] uploading of

[01:17:15] human minds

[01:17:16] into digital substrate

[01:17:19] and then various forms of modification

[01:17:21] of that.

[01:17:23] Maybe the ability to

[01:17:26] using an AI that is

[01:17:28] able to edit each

[01:17:30] synapse

[01:17:31] um

[01:17:33] the ability to then download knowledge

[01:17:36] and skills or reformat your mind in

[01:17:38] different ways. I think there's just

[01:17:40] this huge space of possibilities that

[01:17:42] opens up.

[01:17:45] Um

[01:17:48] a much larger realm of possible modes of

[01:17:51] being

[01:17:53] where hopefully we would get

[01:17:56] the opportunity to sort of charter

[01:17:59] the trajectories out into this much

[01:18:01] larger space where perhaps, you know, if

[01:18:04] the urgency was removed, if we had, say,

[01:18:07] cures for

[01:18:08] all diseases and

[01:18:11] for aging itself and

[01:18:13] AIs that could look after us to make

[01:18:15] sure that we didn't catastrophically

[01:18:17] destroy the world

[01:18:19] um

[01:18:20] maybe then it would make sense to

[01:18:23] slow down a little bit and to sort of

[01:18:26] pursue some path that perhaps eventually

[01:18:28] results in us becoming some sort of

[01:18:30] strange posthuman

[01:18:32] type of being or beings.

[01:18:35] But

[01:18:37] along a path where maybe we sort of

[01:18:39] pause to smell the flowers as as we go

[01:18:42] along. That there might be certain types

[01:18:44] of values that

[01:18:46] are realizable

[01:18:48] with our current human set of capacities

[01:18:51] and then maybe we would want to explore

[01:18:53] those for a bit before then maybe

[01:18:55] gradually expanding

[01:18:58] and upgrading

[01:18:59] our capabilities and then

[01:19:01] eventually that might lead us to grow

[01:19:04] into something

[01:19:06] um

[01:19:07] quite different just as

[01:19:09] like if you have a toddler they will

[01:19:11] eventually grow into something quite

[01:19:13] different like an adult that's in many

[01:19:15] respects almost a different kind of

[01:19:17] entity than a toddler in terms of what's

[01:19:20] going on inside their minds, the kind of

[01:19:21] problems they're dealing with.

[01:19:24] Um and yet

[01:19:27] for the most part we don't think it's

[01:19:28] bad for a toddler to grow up even though

[01:19:30] it means the toddler is no longer there.

[01:19:32] And so

[01:19:33] perhaps similarly with us humans we now

[01:19:35] think of us as being grown-ups and being

[01:19:38] so very mature and big but I think we

[01:19:41] are like babies really. Uh it's just

[01:19:43] kind of

[01:19:45] stunted

[01:19:47] because of our biological constraints so

[01:19:49] we just cease growing up.

[01:19:51] Yeah.

[01:19:51] And have this arrested development at

[01:19:53] age 20 and then we stay hovering for a

[01:19:56] few decades and then

[01:19:58] just as we have started to acquire a

[01:20:00] little bit of knowledge and experience

[01:20:02] our brain starts to rot.

[01:20:04] And it's all erased again. And and maybe

[01:20:06] that could be a way to sort of allow us

[01:20:08] to continue to grow and develop together

[01:20:10] with with our friends and and

[01:20:12] communities.

[01:20:14] Um and it would be exciting to see what

[01:20:16] kinds of

[01:20:17] level of maturity and like

[01:20:21] self-realization that would be possible

[01:20:23] if we could continue you know, for many

[01:20:25] hundreds of years and then maybe

[01:20:27] eventually you could imagine adding

[01:20:29] extra neurons and extra

[01:20:32] memory and extra forms of vitality and

[01:20:35] new emotional responses and so forth.

[01:20:38] I I think we just haven't seen uh

[01:20:40] nothing yet in terms of what's possible.

[01:20:42] Yeah, I I I totally agree with you.

[01:20:44] There is a science fiction book written

[01:20:46] by Arthur C. Clarke years ago called

[01:20:49] Childhood's End. Basically, the end of

[01:20:52] the childhood of the species called

[01:20:55] humankind, you know, and this may well

[01:20:57] be the moment where we're starting on

[01:21:00] the trajectory of ending the childhood

[01:21:02] of the species and it's exciting to see

[01:21:04] where it goes. All the same though, just

[01:21:05] to paraphrase what you said earlier,

[01:21:07] it's both exciting and terrifying and

[01:21:10] you have to perhaps have a fatalistic

[01:21:12] outlook, do everything that we can to

[01:21:13] shape the betterest future we could have

[01:21:16] and we'll just see where it all goes.

[01:21:19] So, Nick Bostrom, thank you so much for

[01:21:21] joining us.

[01:21:22] Oh, it's fun.

[01:21:23] exciting conversation and I look forward

[01:21:26] to seeing where this all goes.

[01:21:28] Yeah, it'll be an exciting ride if

[01:21:30] nothing else.

[01:21:30] Absolutely. Thank you.

[01:21:33] Thanks.