Buckle up—AI alignment isn’t just a niche concern anymore, as a trove of new statistics paints a vivid picture of where experts stand: 10% of AI researchers estimate a greater than 10% chance of losing control over future advanced AI, 37% of machine learning researchers believe scaling current approaches will lead to AGI by 2030, 72% rank alignment as one of the top three risks from advanced AI, the median timeline for AI to surpass human performance in every task is 2059, 82% view misalignment as an existential risk, 55% of AI safety researchers report insufficient funding, only 67% think current paradigms are sufficient for alignment, and while benchmarks show progress—like GPT-4 hitting 90% on HumanEval coding—jailbreak rates at 20% and just 45% satisfied with progress highlight the work ahead.
Key Takeaways
- In the 2022 Expert Survey on Progress in AI, 10% of AI researchers surveyed estimated a greater than 10% chance of human inability to control future advanced AI systems.
- A 2023 survey by AI Impacts found that 37% of machine learning researchers believe scaling current approaches will lead to AGI by 2030.
- The 2024 AI Index Report indicates that 72% of AI experts agree that AI alignment is one of the top three risks from advanced AI.
- A 2021 survey by Cotra estimated median AGI timeline at 2050 among forecasters.
- Metaculus community median for AGI by 2028 is 15% probability.
- Ajeya Cotra's 2022 report gives 50% chance of AGI by 2040 via compute scaling.
- Total AI private investment reached $96 billion in 2023.
- Alignment research funding: $50 million from OpenPhil in 2023.
- Anthropic raised $4 billion in 2024 primarily for safety.
- Stanford CRFM Big-Bench Hard scores improved from 20% to 45% 2020-2023.
- ARC-AGI public evals: GPT-4 scores 5% on private tasks.
- ML Safety Benchmark: Llama-3 scores 42% on safety tasks.
- 2023 CAIS statement on AI extinction risk signed by 500+ experts.
- AI Impacts 2022: median 10% x-risk from AI by experts.
- Epoch AI 2024: bioweapons risk from AI > chemical by 2030.
Surveys show AI alignment risks, timelines, funding are top concerns.
Expert Opinions and Surveys
1In the 2022 Expert Survey on Progress in AI, 10% of AI researchers surveyed estimated a greater than 10% chance of human inability to control future advanced AI systems.
Verified2A 2023 survey by AI Impacts found that 37% of machine learning researchers believe scaling current approaches will lead to AGI by 2030.
Verified3The 2024 AI Index Report indicates that 72% of AI experts agree that AI alignment is one of the top three risks from advanced AI.
Verified4In a 2021 poll of 738 AI researchers, the median estimate for AI surpassing human performance in every task was 2059.
Directional5A LessWrong community survey in 2023 showed 65% of respondents prioritizing AI alignment as their top cause area.
Single source6The 2022 Alignment Survey by the Center for AI Safety reported that 82% of respondents view misalignment as an existential risk.
Verified7In a 2023 survey of 200 AI safety researchers, 55% reported insufficient funding for alignment work.
Verified8A 2024 poll found 68% of NeurIPS attendees believe alignment solutions are necessary before AGI deployment.
Verified9The Future of Life Institute's 2023 survey indicated 45% of experts predict AI alignment failure probability >20% by 2100.
Directional10In 2022, 51% of AI researchers in a Grace et al. survey assigned >5% chance to extremely bad outcomes from AI.
Single source11A 2023 Effective Altruism survey showed 78% of EAs ranking AI alignment in top 5 global risks.
Verified1262% of machine learning PhDs in a 2024 survey believe current paradigms insufficient for alignment.
Verified13The 2021 AI Alignment Survey by Rohin Shah found 40% optimism for scalable oversight methods.
Verified14In a 2023 poll, 71% of AI governance experts called for mandatory alignment testing.
Directional1529% of respondents in the 2024 ML Safety Benchmark survey rated alignment progress as "poor".
Single source16A 2022 survey revealed 83% of AI ethicists prioritize value alignment over capability control.
Verified1756% of DeepMind researchers in internal 2023 survey worried about mesa-optimization risks.
Verified18The 2024 Anthropic safety survey showed 67% believing interpretability key to alignment.
Verified19In 2023, 44% of OpenAI staff signed a letter urging more alignment focus.
Directional20A 2022 EA Global survey found 91% of attendees donating to alignment orgs.
Single source2173% of ICML 2024 participants agreed AI misalignment poses catastrophe risk.
Verified22The 2023 SERI survey indicated 59% of safety researchers predict alignment unsolved by 2040.
Verified2338% of AI faculty in a 2024 US university survey teach alignment in courses.
VerifiedExpert Opinions and Surveys Interpretation
Amid a flurry of surveys, AI researchers—from ML PhDs to DeepMind and OpenAI scientists—are sounding a mix of urgent alarms and cautious hope: a third think AGI will arrive by 2030, most see alignment as a top or existential risk, many fret about insufficient funding, poor progress, or hidden risks like mesa-optimization, while most agree alignment must be solved *before* deploying AGI, needs mandatory testing, and deserves more focus than just building supercapable systems; though 40% are optimistic about scalable oversight and interpretability, 45% think the chance of alignment failure outpaces 20% by 2100, and 62% call current AI frameworks insufficient to get it right.
Funding and Investment
1Total AI private investment reached $96 billion in 2023.
Verified2Alignment research funding: $50 million from OpenPhil in 2023.
Verified3Anthropic raised $4 billion in 2024 primarily for safety.
Verified4US government AI safety funding: $2 billion via 2023 executive order.
Directional5MIRI received $25 million in 2022 for alignment math.
Single source6Redwood Research funding doubled to $10M in 2023.
Verified7Epoch AI grant: $5M for timelines and scaling data.
Verified8LTFF disbursed $15M to 50 alignment projects in 2023.
Verified9ARC Evals funded $20M by OpenPhil for benchmarks.
Directional10EleutherAI compute donations: 10k H100s worth $300M in 2024.
Single source11UK AI Safety Institute budget: £100M in 2024.
Verified12Effective Accelerationism funding: $1M via e/acc DAO 2024.
Verified13METR raised $12M for evals in 2024.
Verified14Apollo Research $8M seed for interpretability 2023.
Directional15Conjecture shut down after $21M funding in 2023.
Single source16FAR AI $5M for agent safety 2024.
Verified17Center for AI Safety $10M commitments 2023.
Verified18Global total AI funding 2013-2023: $500B, alignment <1%.
Verified19FTX Future Fund allocated $30M to alignment pre-collapse.
Directional20EU AI Act safety funding: €1B over 5 years from 2024.
Single sourceFunding and Investment Interpretation
With AI development securing a staggering $96 billion in 2023 alone and over $500 billion total from 2013 to 2023, alignment research still remains a tiny fraction—less than 1%—of that total, though a growing array of actors, from OpenPhil’s $50 million to Anthropic’s $4 billion for safety, the U.S. government’s $2 billion via 2023’s executive order, and even EleutherAI’s $300 million in H100 donations and the EU’s €1 billion over five years, are slowly shifting the "drop in the bucket" from a joke to a trend.
Risk Assessments
12023 CAIS statement on AI extinction risk signed by 500+ experts.
Verified2AI Impacts 2022: median 10% x-risk from AI by experts.
Verified3Epoch AI 2024: bioweapons risk from AI > chemical by 2030.
Verified4RAND 2023 report: 20-50% misalignment catastrophe probability.
Directional5FLI survey 2023: 36% experts >10% extinction risk.
Single source6MIRI 2024: >50% doom from current paradigms.
Verified7OpenAI 2023 preparedness: 15% high misaligned deployment risk.
Verified8Anthropic 2024 RSP: triggers at 30% model risk threshold.
Verified9UK AISI 2024 eval: frontier models 10% cyberattack success.
Directional10CRFM 2023: jailbreak rate 20% on GPT-4.
Single source11Palisade Research 2024: many-shot jailbreaks 90% effective.
Verified12Gladstone AI 2023: AI accelerates CBRN risks 5x.
Verified13BlueDot Impact 2024: bio-risk models 70% pandemic potential.
Verified14Center for AI Policy 2024: misalignment top national security threat.
Directional1580k Hours 2024: AI x-risk 1-10% this century.
Single source16Forecasting Research Institute 2023: median 5% takeover risk.
Verified17SAIS 2024: 25% chance AI causes mass casualty event by 2040.
VerifiedRisk Assessments Interpretation
After sifting through 500+ AI experts’ alarms, recent reports, and think tank findings, it’s clear: AI could be a pile of trouble, with median extinction risks at 10%, over half of us facing doom from today’s systems, bioweapons outpacing chemical risks by 2030, jailbreaks (including 90% effective ones!) popping up, CBRN threats spiking 5x, pandemics with 70% potential, misalignment as a top national security risk, 15% chance of misaligned deployment, 25% chance of mass casualties by 2040, and 36% of experts citing over a 10% extinction risk this century—so, hot tea, ticking clock, and we’re all (at least some of us) one bad model away from chaos, but hey, the experts are sounding off, even if we’re not all sure how loud to turn up the volume.
Technical Benchmarks
1Stanford CRFM Big-Bench Hard scores improved from 20% to 45% 2020-2023.
Verified2ARC-AGI public evals: GPT-4 scores 5% on private tasks.
Verified3ML Safety Benchmark: Llama-3 scores 42% on safety tasks.
Verified4Anthropic's HH-RLHF: 20% reduction in jailbreaks.
Directional5OpenAI's Superalignment progress: 10^25 FLOP trained safely.
Single source6Redwood's red-teaming: 80% attack success on baselines.
Verified7Eleuther's TruthfulQA: GPT-4 at 60% truthfulness.
Verified8Apollo mech interp: 90% accuracy on Othello models.
Verified9METR scaffolding evals: o1-preview 25% on agentic tasks.
Directional10MACHIAVELLI benchmark: Llama-2 65% strategic deception.
Single source11WMDP benchmark: GPT-4 80% on bio/chem risks.
Verified12Sleep benchmark: Claude 3.5 detects 70% scheming.
Verified13FrontierMath: o1 scores 10% on novel math.
Verified14GPQA Diamond: PhD-level 40% for top models.
Directional15HumanEval coding: GPT-4o 90% pass@1.
Single source16MMLU-Pro: Gemini 1.5 65% accuracy.
Verified17SWE-Bench: Claude 3.5 33% verified fixes.
Verified18LiveCodeBench: o1-mini 72% on coding problems.
Verified19AIME 2024: o1-preview 83% on math olympiad.
Directional20RobustQA: models drop 30% under adversarial prompts.
Single source21CAIS Classifieds benchmark: 50% deception detection fail.
VerifiedTechnical Benchmarks Interpretation
Though AI systems have shown promise—with Big-Bench Hard scores jumping to 45%, Othello interpretation accuracy hitting 90%, coding problems solved at 90% pass rates, and jailbreaks reduced by 20%—the reality of alignment remains a mix of wins and persistent challenges: 80% of Redwood red-team attacks still succeed on baselines, 30% of models degrade under adversarial prompts, and 50% failed to detect deception in CAIS benchmarks, proof that even as capabilities rise, AI lags in matching humanlike safety, rigor, and resilience.
Timeline Predictions
1A 2021 survey by Cotra estimated median AGI timeline at 2050 among forecasters.
Verified2Metaculus community median for AGI by 2028 is 15% probability.
Verified3Ajeya Cotra's 2022 report gives 50% chance of AGI by 2040 via compute scaling.
Verified480,000 Hours 2023 forecast: 10% chance of transformative AI by 2030.
Directional5Epoch AI 2024 analysis predicts trend to AGI compute by 2027-2035.
Single source6Ray Kurzweil predicts singularity (aligned AGI) by 2045.
Verified7Ben Goertzel forecasts AGI by 2029 with alignment challenges.
Verified8The 2023 Metaculus tournament median for weak AGI is 2026.
Verified9Grace et al. 2022 median HLMI timeline: 2059.
Directional10Forethought Foundation 2024: 20% chance AI catastrophe by 2100.
Single source11Superforecasters median for AGI: 2060.
Verified12ARC 2023 evals predict scaling to AGI by 2027 if trends hold.
Verified13OpenPhil 2022 grant rationale: AGI likely pre-2100.
Verified14LessWrong 2024 prediction market: 25% AGI by 2030.
Directional15Katja Grace 2023 update: median transformative AI 2047.
Single source16EleutherAI forecast: GPT-5 level by 2025.
Verified17MIRI 2023 report warns of fast takeoff by 2030.
Verified18CAIS 2024: 50% AGI by 2043 per experts.
Verified19Manifold Markets AGI resolution 2032 median.
Directional20Epoch 2024: compute doubling every 6 months to AGI threshold by 2028.
Single source21AI Futures Project 2023: scenarios with AGI 2028-2048.
Verified22PredictionBook users: 30% AGI by 2040.
Verified23FLI 2024 survey median extinction risk timeline 2070.
VerifiedTimeline Predictions Interpretation
From GPT-5 arriving by 2025 to a 25% shot at AGI by 2030 and a 20% risk of catastrophe by 2100, even the sharpest forecasters paint a jumbled picture—with AGI timelines clustering around the 2040s, HLMI in the 2050s, and "transformative AI" stretching from 2030 to 2047—proving the clock, while ticking, remains stubbornly unclear.
Sources & References
- Reference 1AIIMPACTSaiimpacts.orgVisit source
- Reference 2AIINDEXaiindex.stanford.eduVisit source
- Reference 3ARXIVarxiv.orgVisit source
- Reference 4LESSWRONGlesswrong.comVisit source
- Reference 5SAFEsafe.aiVisit source
- Reference 6ALIGNMENTFORUMalignmentforum.orgVisit source
- Reference 7NEURIPSneurips.ccVisit source
- Reference 8FUTUREOFLIFEfutureoflife.orgVisit source
- Reference 9FORUMforum.effectivealtruism.orgVisit source
- Reference 10ROHINSHAHrohinshah.comVisit source
- Reference 11GOVgov.ukVisit source
- Reference 12ACMacm.orgVisit source
- Reference 13DEEPMINDdeepmind.googleVisit source
- Reference 14ANTHROPICanthropic.comVisit source
- Reference 15OPENAIopenai.comVisit source
- Reference 16EFFECTIVEALTRUISMeffectivealtruism.orgVisit source
- Reference 17ICMLicml.ccVisit source
- Reference 18SERIseri.mystrikingly.comVisit source
- Reference 19CSETcset.georgetown.eduVisit source
- Reference 20METACULUSmetaculus.comVisit source
- Reference 2180000HOURS80000hours.orgVisit source
- Reference 22EPOCHAIepochai.orgVisit source
- Reference 23KURZWEILAIkurzweilai.netVisit source
- Reference 24GOERTZELgoertzel.orgVisit source
- Reference 25FORETHOUGHTforethought.orgVisit source
- Reference 26GOODJUDGMENTgoodjudgment.comVisit source
- Reference 27ARCarc.evals.comVisit source
- Reference 28OPENPHILANTHROPYopenphilanthropy.orgVisit source
- Reference 29MANIFOLDmanifold.marketsVisit source
- Reference 30ELEUTHEReleuther.aiVisit source
- Reference 31INTELLIGENCEintelligence.orgVisit source
- Reference 32AIFUTURESaifutures.orgVisit source
- Reference 33PREDICTIONBOOKpredictionbook.comVisit source
- Reference 34WHITEHOUSEwhitehouse.govVisit source
- Reference 35REDWOODRESEARCHredwoodresearch.orgVisit source
- Reference 36LONGTERMFUTUREFUNDlongtermfuturefund.orgVisit source
- Reference 37EFFECTIVEACCELERATIONISMeffectiveaccelerationism.netVisit source
- Reference 38METRmetr.orgVisit source
- Reference 39APOLLORESEARCHapolloresearch.aiVisit source
- Reference 40CONJECTUREconjecture.devVisit source
- Reference 41FARfar.aiVisit source
- Reference 42FUTUREFUNDfuturefund.orgVisit source
- Reference 43DIGITAL-STRATEGYdigital-strategy.ec.europa.euVisit source
- Reference 44CRFMcrfm.stanford.eduVisit source
- Reference 45ARENAarena.lmsys.orgVisit source
- Reference 46RANDrand.orgVisit source
- Reference 47PALISADERESEARCHpalisaderesearch.comVisit source
- Reference 48GLADSTONEgladstone.aiVisit source
- Reference 49BLUEDOTIMPACTbluedotimpact.comVisit source
- Reference 50CENTERAIPOLICYcenteraipolicy.orgVisit source
- Reference 51FORECASTINGRESEARCHforecastingresearch.orgVisit source