π_π
@m-j-r
1366 Following
27853 Followers
0 reply
0 recast
4 reactions
rebuttal to Daniel Kokotajlo's rebuttal
https://x.com/DKokotajlo/status/1943802695464497383
tl;dr Daniel, being the claimant, must address the public's continued ability to adapt, just as he must elaborate on the supposed monolith's ability to surpass bottlenecks that are practically discovered so far.
the onus from AI 2027 does not fall to the responsibility of any critic describing the alternative.
βΈ»
V - Individuals need to be equipped with locally-running AI that is explicitly loyal to them
D - In the Race ending of AI 2027, humanity never figures out how to make AIs loyal to anyone. OpenBrain doesn't slow down, they think they've solved the alignment problem but they haven't. Maybe some academics or misc minor companies in 2028 do additional research and discover e.g. how to make an aligned human-level AGI eventually, but by that point it's too little, too late (and also, their efforts may well be sabotaged by OpenBrain/Agent-5+, e.g. with regulation and distractions.
βΈ»
if every household has a dirty qubit, everyone is aligned to error correction, and the neighborhoods will cross-sample error correction and nothing more, as latency permits. it would be needlessly costly to anthropomorphize quantum error correction. other economic applications also point to ASICs, like industrial robotic arms vs sentient humanoids. economy of scale, not economy of impression, is what consumers want to see on the price tag (consider modern Chinese dark factories).
βΈ»
V - Defense technologies should be more of the "armor the sheep" flavor, less of the "hunt down all the wolves" flavor.
D- This might be the only item on this list that I disagree with. I agree that given a choice between armoring the sheep and hunting down the wolves, we should prefer armoring the sheep. But sometimes we simply don't have a choice. E.g. our solution to murder is to hunt down murderers, not to give everyone body armor and so forth so that they can't be killed, because that simply wouldn't be feasible.
βΈ»
violent crime is often solved by confession and the deterrence of detectors like gunfire locators. where there's emergent need to find victims or perpetrators, an alert is broadcasted and superintelligence of the crowd takes over. we actually do try to give everyone abstract armor, such as bartenders and dispatchers looking out for coded messages, and instructions on leaving behind a canary message before disappearing into a sparse secondary location. criminals are opportunists, and the risk of imperfect secrecy within immutable record is the greatest deterrent
βΈ»
V - it actually feels implausible that we don't have a wearable device that can bio-print and inject things into you in real time to keep you safe even from arbitrary infections (and poisons).
D - [what if superintelligences] used their lead to prevent the creation of rivals e.g. by political maneuvering to do a merger and/or simply by driving rival companies out of business with superior products? Then there'd be no one developing these wearables, except for the OpenBrain ASI's themselves and their derivatives, and of course they'd make sure to sabotage them insofar as that would help their long-term plans.
βΈ»
as I pointed out before (https://farcaster.xyz/m-j-r.eth/0xb71e9e64), there's already public demand for right-to-repair, and there are open standards & controls for biotech that isn't just fictional beakers mixed together. there's already backlash to HIPAA anti-competition, and we have ~5+ decades of guerilla drug manufacture. this field can't be forced into a monolith without a common public understanding of its abuses (e.g. anthrax & covid leaks).
βΈ»
V - If the world's strongest AI can turn the world's forests and fields into factories and solar farms by 2030, the world's second-strongest AI will be able to install a bunch of sensors and lamps and filters in our buildings by 2030.
D - I'm predicting that power will have concentrated/consolidated too much by this point. E.g. in the slowdown ending the US companies merge. Also, the speed of takeoff is such that e.g. a six-month lead is probably too big of a lead
βΈ»
even in the leading example of consolidation like China procuring several terawatts over many years, it's not dozens of terawatts in six months for leaps of capability. otherwise, if leap in architectural design permits leaps of capability, this becomes an issue of proliferation, therefore back to the same issue of guerilla/outlaw ecosystem that evolves its own tamper-evident security as needed. furthermore, immutable record broadcasts beyond any actor's physical ability to degrade broadcast signal. so superintelligence, today, can consider the risk of irrevocably telegraphing their attacks on public systems.
βΈ»
V - ...to have access to good info defense tech. This is relatively more achievable within a short timeframe
D - how are we going to get slightly less wildly superintelligent analyzers to help out decision-makers, so that we don't need to blindly trust that the even-more-wildly superintelligent super-persuaders in the leading US AI project are trustworthy? Answer: We aren't
βΈ»
pessimistically, super-persuasion is the easiest capability that humanity has ever realized. we typically adapt to things like video platform algorithms and internet addiction by freely criticizing them in public and private, even while burdened with these issues. strengthening our defense is a matter of freeing and amplifying public signal against the totalitarianism we already experience and reject. we don't need yet another company when Solzhenitsyn can just self-publish on censorship-resistant platforms.
βΈ»
V - The success of the kinds of countermeasures described above, especially the collective measures that would be needed to save more than a small community of hobbyists, rests on three preconditions:
D - I agree for weak definitions of success (i.e. making a total-victory-decapitation strike not happen) but disagree for strong definitions of success (i.e. preventing Consensus-1 from winning the war). To prevent Consensus-1 from winning the war it's not enough that e.g. France's power grid and network are resistant to superintelligent hacking. France has to be able to beat Consensus-1's military, which at that point is a huge force of robots/drones/etc. produced in both the US and China, the world's largest and most advanced economies by a lot thanks to the ongoing industrial explosion.
βΈ»
it should be clear by now that central armament logistics was a 20th century edge, but is decreasingly true in the 21st century. countries import drones that are neither Chinese nor American, moreover these don't require either set of economic rails. Ukraine and Israel have shown that detection of drone transportation is unreliable. the markets for the raw materials may be concentrated, but it's a stretch that the global market could be shrunk without geopolitical blowback and many theaters/fronts opening up on that alone. after all, this precipitated involvement in 20th century global conflicts. on top of all this, there's advancements like fiber optic drones, coupled with canary messages and broadcast, immutable evidence. diasporic insurgency is more equipped than ever before. war is not the monolith being imagined in this scenario.
βΈ»
V - The AI 2027 scenario implicitly assumes that the capabilities of the leading AI (Agent-5 and then Consensus-1), rapidly increase, to the point of gaining godlike economic and destructive powers, while everyone else's (economic and defensive) capabilities stay in roughly the same place. This is incompatible with the scenario's own admission (in the infographic) that even in the pessimistic world, we should expect to see cancer and even aging cured, and mind uploading available, by 2029.
D - I don't see the contradiction. We didn't say this one way or another iirc, but my headcanon is that in 2028, the leading AIs + AI companies basically work to gobble up, partner with, or squash their competitors. In the slowdown ending the various US projects merge. In the race ending we don't really talk about it but I imagine a merger would happen too. So yeah, lots of amazing technologies get developed over the course of 2028 and 2029, but the entities doing the developing are almost all OpenBrain or DeepCent AIs (or derivatives), all working together towards misaligned goals. Massive concentration of power in these two power centers, basically, such that if they can make a deal with each other, the whole rest of the world gets cut out.
βΈ»
IP is already leaky, if not entirely copyleft. there's never been a runaway megacorporate power that has successfully taken over world government, even when considered the frightening reach of firms like Blackrock, Meta, or Palantir. the public already has the ability to tokenize any mode of production onchain, including systems like Bittensor and PrimeIntellect. any claim of runaway consolidation needs to explain the inevitability & infallibility of regulatory capture...of the entire internet. and it needs to address the commonplace defection from firms that we already observe, combined with the enormous pools of third-world capital that will incorporate in a race condition. there's a lot of optionality, even as SOTA keeps getting commoditized. nobody is at a standstill, and everyone under the sun wants lower costs. clearly, the global public is not settling for an oligopoly of overpriced APIs, and many innovations manifest where the resources are constrained below the OpenBrain/Deepcent line, including Moravec's paradox as applied to embodied intelligence. the claim of two datacentered monoliths is extremely unsound.
βΈ»
ultimately, I agree with Vitalik's d/acc approach, as much as the free global market can bear. 0 reply
0 recast
1 reaction
1 reply
0 recast
2 reactions
1 reply
0 recast
3 reactions
1 reply
0 recast
5 reactions
0 reply
0 recast
0 reaction
1 reply
0 recast
3 reactions
1 reply
0 recast
1 reaction
1 reply
0 recast
5 reactions
0 reply
0 recast
1 reaction
1 reply
1 recast
1 reaction
0 reply
0 recast
0 reaction
0 reply
0 recast
1 reaction
2 replies
0 recast
5 reactions
no Volcker, microtransistors and personal computing would have reached a different economy of scale.
unlikely, but the proliferation of digital scientific instrumentation may have still discovered fracking and natgas, so altogether the calculus for domestic manufacturing would be too good to offshore.
intercity rail may have scaled differently, so suburban development may have scaled differently. immigration policy would have needed to expand further. mass media would have differentiated further, internet would be even more convenient, and neural networks would have been further explored.
meanwhile, global commerce would go brrr as the reserve currency would be that much more attractive.
although, all of this may still have never happened because history, like nature, is a fickle beast. 2 replies
2 recasts
8 reactions
0 reply
0 recast
0 reaction
1 reply
0 recast
4 reactions
0 reply
0 recast
3 reactions
1 reply
1 recast
2 reactions
0 reply
0 recast
7 reactions