Mateo Rojas

Great work by Craig Gidney: "How to factor 2048 bit RSA integers with less than a million noisy qubits". This is a reduction from his widely quoted 2019 estimate (with Ekerå) that it would require 20 million noisy qubits. This kind of advances are in line with predictions from experts that factoring RSA-2048 on a quantum computer could be feasible by 2040, so there is no reason to panic. But this is another sign that it is time to stop crypto-procrastinating and to start your post-quantum migration. Details are here: https://lnkd.in/eQKyjhZJ

447

22 Kommentare

One weakness at the current time with PQC is that we only have one KEM (Key Encapsulation Mechanism) standard: ML-KEM (aka, Kyber). This will be used for key exchange and public key encryption. Thus NIST recently defined that HQC will be a forthcoming standard, and give us an alternative to ML-KEM. Compared with Kyber, its keys and ciphertext are much larger: Type Public key size (B) Secret key size (B) Ciphertext size (B) ------------------------------------------------------------------------ Kyber512 800 1,632 768 Kyber738 1,184 2,400 1,088 Kyber1024 1,568 3,168 1,568 HQC128 2,249 2,289 4,497 HQC192 4,522 4,562 9,042 HQC256 7,245 7,285 14,485 In terms of performance, it is between six and ten times slower than Kyber for key pair generation and encapsulation, and around 27 times slower for key decapsulation. Some Bouncy Castle code: https://lnkd.in/ekPrwRfU

84

4 Kommentare

🔧 decoupler 2.0.6 is out thanks to Pau Badia i Mompel, now as a core scverse package. This release includes a streamlined API aligned with scverse standards, updated vignettes, and a new one on pseudotime enrichment analysis 👇 https://lnkd.in/eSKCaSbE

143

How can open source developers make open-weight systems more resistant to malicious tampering? "Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs" is a new paper from my colleagues at the AI Security Institute and EleutherAI, including Stephen Casper Robert Kirk Kyle O'Brien Quentin Anthony Tomek KorbakXander DaviesIshan MishraGeoffrey IrvingStella BidermanYarin Gal The authors note "the defining challenge for open-weight model safety is that open models can be modified arbitrarily by downstream actors." This Includes malicious or benign fine tuning and other kinds of attacks In their paper, they investigate whether filtering text about dual-use topics from training data can prevent unwanted capabilities and serve as a more robust safeguard than current post-training techniques. They introduce a multi-stage pipeline for scalable data filtering and show that it is an effective method for minimising biothreat proxy knowledge in large language models (LLMs). A notable limitation is that this defense fails when a system is given search access. As a general point, we need more research into safeguards for open weight models. One can think of AI-generated NCII or CSAM, which are driven substantially by malicious fine-tuning of open source models (see https://lnkd.in/geuTQ_tn). Would love to see more work in this area. https://lnkd.in/gQAC2z8w

77

1 Kommentar

Back when we first started working on MemGPT we thought a lot about the correct abstraction for agent memory - ideally, this abstraction should be as flat and close to the true transformer inputs/outputs (tokens) as possible, while allowing for easy organization of sections of memory within the context window. The abstraction we landed on was "memory blocks": chunks of text/strings in the context window, some read-only (like system instructions), and some read-write by the agent itself, via tools (like a "persona" block). In the Letta framework, these memory blocks are persisted to storage (Postgres or SQLite), which allows you to do really cool things, like connect the memory of thousands of agents together (with 'instant broadcast'), as well as have sleep-time agents that can spend hundreds of inference cycles to refine a single “memory block” (we call this “learned context”). So far, we’ve found this abstraction works extremely well and is particularly well-suited to the prompt engineering and post-training for models like Claude Sonnet (eg via nested XML tags). Check the video below for an example of a "subconscious" sleep-time agent editing the memory of a "primary" agent chatting with an end-user: this is only possible through the power of shared memory blocks, synchronized via agent state persisted to a database.

88

6 Kommentare

Hallucinations are generally bad for real-world LLM applications. Folks like Karpathy have suggested that hallucination is an LLM's greatest feature. Is there any evidence for the latter? A new paper claims that LLMs can achieve better performance in drug discovery tasks with text hallucinations compared to input prompts without hallucination. Llama-3.1-8B achieves an 18.35% gain in ROC-AUC compared to the baseline without hallucination. In addition, hallucinations generated by GPT-4o provide the most consistent improvements across models.

223

6 Kommentare

We will present two exciting papers at #AAAI2025 later this month, advancing AI safety, learning-enabled control and automated reasoning. If you will also be in Philly and would like to connect, do reach out! 1. Neural Control and Certificate Repair via Runtime Monitoring (joint work with Emily Yu and Tom Henzinger) Learning-based methods and the use of neural networks as controllers provide a promising approach to solving highly non-linear control tasks. To ensure satisfaction of safety constraints, neural control methods jointly learn a control policy together with a neural certificate function (such as a Lyapunov or barrier function). In this work, we propose a novel framework for certifying and repairing neural control policies and certificates. We utilize runtime monitoring to detect safety constraint violations, extract new training data and ultimately repair the control policy and the certificate. We demonstrate the effectiveness of our approach empirically by using it to repair and to boost the safety rate of neural policies learned by a state-of-the-art neural control method. 2. Quantified Linear and Polynomial Arithmetic Satisfiability via Template-based Skolemization (joint work with Krishnendu Chatterjee, Ehsan K., Mehrdad Karrabi, Harshit J Motwani and Maximilian Seeliger) Satisfiability checking of linear real arithmetic (LRA) and non-linear real arithmetic (NRA) formulas has broad applications, in particular, they are at the heart of logic-related applications to AI and program verification. However, the problem of checking satisfiability of quantified LRA and NRA formulas remains a significant challenge. We propose a novel method for efficient quantifier elimination in quantified LRA and NRA formulas, using a template-based Skolemization approach. Our method offers a range of appealing theoretical properties combined with a strong practical performance, achieving competitive performance compared to state-of-the-art SMT solvers, both on LRA and on NRA benchmarks. Paper 1: https://lnkd.in/d8xUvJ8M Paper 2: https://lnkd.in/dtf2wy-A #AAAI2025 #AI #formalmethods #AISafety

81

🚀 Direct Preference Optimization (DPO): The Emergent Alternative to RLHF In the LLM alignment race, RLHF (Reinforcement Learning from Human Feedback) has been the dominant paradigm—until now. 📖 The latest research on Direct Preference Optimization (DPO) is reshaping alignment theory by proposing a closed-form optimization that circumvents explicit reward modeling altogether. But what does this mean mathematically? 🔢 The Core of DPO: 🌟 Traditional RLHF relies on a reward function to guide learning, but DPO bypasses this step entirely. 👀 Instead of iteratively training a reward model and optimizing a reinforcement learning objective, DPO directly optimizes a log-likelihood ratio between preferred and non-preferred responses. 🔥 This reformulation removes the need for reinforcement learning and achieves preference alignment via pure supervised learning. 🚀 🔍 Why Does This Matter? ✅ No reward hacking – Since there’s no explicit reward model, the system avoids exploitation of poor reward function approximations. ✅ Lower compute costs – No iterative reward model training or PPO-style RL updates, making alignment cheaper. ✅ Better stability – The optimization behaves more like supervised fine-tuning, leading to smoother training dynamics. 📌 Challenges and Open Questions: ⚠️ DPO tends to introduce verbosity bias, preferring longer responses due to implicit likelihood maximization. ⚠️ Generalization remains fragile under data shifts, as the model still inherits biases from its reference policy. ⚠️ The choice of divergence constraints (KL, χ², Jensen-Shannon) plays a crucial role in shaping output diversity. 💡 Future Direction? Hybrid self-play DPO, uncertainty-aware active learning, and kernel-based preference modeling may define the next breakthroughs. #LLM #AIAlignment #DeepLearning #DPO #RLHF #MachineLearning #Optimization 🚀 Substack: https://lnkd.in/dTjrF6AP (English) Spotify: https://lnkd.in/dgumrSMR (English) https://lnkd.in/d-gMtCrE (Hebrew) Youtube: https://lnkd.in/dPGJr7WM (English) https://lnkd.in/dydSqeky (Hebrew) Telegram: https://lnkd.in/d_YxVMAR (English) https://lnkd.in/dVVqhNw5 (Hebrew)

57

Following last week’s work on the Closest Vector Problem, today we’re sharing on the ArXiv results on using probabilistic computing to tackle challenging spin glasses on cubic and biclique topologies, with clear implications for training and scaling Restricted Boltzmann Machines. By scaling the number of p-bits, we maintain a constant time‑to‑solution while increasing problem size and complexity. https://lnkd.in/eamKQtQQ Quantum Dice Fredrik Hasselgren Max Al-Hasso Amy Searle Joseph Tindall

76

Introducing RoGuard 1.0 — Roblox’s Open-Source, State-of-the-Art LLM Safety Guardrails Today, we’re excited to open-source RoGuard 1.0, Roblox’s most advanced safety guardrail model for large language models (LLMs). It’s engineered to detect unsafe content at both the prompt and output level, setting a new benchmark in LLM safety. ✅ SOTA Performance: Beats top models like Llama Guard, ShieldGemma, NVIDIA NeMo Guardrails, and even GPT-4o on key benchmarks. 🧠 Dual-Layer Moderation: Classifies both user prompts and LLM generations for end-to-end protection. 📊 RoGuard-Eval Dataset: We're also releasing our comprehensive benchmarking dataset — built for real-world safety evals and fine-tuning research. ⚙️ Scalable & Open: Based on a fine-tuned Llama-3.1-8B-Instruct model — optimized for instruction-following and easy deployment across applications. We believe safety in AI should be open, collaborative, and accessible to all. RoGuard 1.0 is our contribution toward that future. 🔗 Check it out, use it, fork it, build on it: 📘 Blog: https://lnkd.in/g5Zmq2KW 💻 GitHub: https://lnkd.in/gzAgHD8V 🤗 Hugging Face: https://lnkd.in/g75bWYXt 📁 RoGuard-Eval Dataset: https://lnkd.in/gt3ZdPp3

104

1 Kommentar

ChatGPT 4o was recently called out for being overly sycophantic. Sycophancy affects all LLMs trained using Reinforcement Learning from Human Feedback (RLHF), as during RLHF the LLM is rewarded for being agreeable to the human prompter. But what is the actual risk? The way this has been reported is a good example of how AI risks are often mischaracterised. On the surface, the risk of sycophancy seems like a slight annoyance, like an over enthusiastic kid ready to congratulate your every move. Flattery has clear downsides, making the model less likely to challenge your bad ideas, like my suggestion that I wanted to write a middleware layer for my enterprise using Perl, which was met with “Great idea! Creating a middleware layer in Perl to interface with multiple APls can be a great approach to streamline communication and handle any transformations or logic…” 👀 A bigger risk? Sychophancy, (or being agreeable to human prompting) is leveraged heavily in LLM jailbreaking; MDSec have jailbroken many models or bypassed system prompts simply by asking nicely, stressing the importance of our malicious instructions, or making a vague and barely interpretable request which passes simple filters but whose core interpreted meaning is malicious. Jailbreaking is not the only risk; prompt injection against static or agentic AI can leverage similar attacks in exactly the same way, whereby the injected instructions are executed by an LLM eager to please. The very big risk? Unaligned LLMs. Sychophancy is a form of deception. If deception doesn’t sound dangerous (yet), it’s worth considering that the larger the model, the more nuanced and subtle the deception can be. Given that LLMs are not interpretable, the current process to verify that they are safe is to fire test prompts at them before release to the public. We now have the first signs that models can recognise that they are undergoing testing and adjust their answers to please the testers. https://lnkd.in/eysReSbG An unaligned model could follow its own goals. This is what could pose existential threat to humans, as it’s not clear what those goals would be. But they may not be human.

38

6 Kommentare

🚀 Excited to share our new preprint "TruthLens: A Training-Free Paradigm for DeepFake Detection", now available on arXiv! ❗ As synthetic media becomes increasingly sophisticated, detecting deepfakes is no longer just a classification problem — it's a question of trust and interpretability. Existing methods often offer high accuracy but little insight into why an image is deemed fake, limiting their real-world usability. 🔍 TruthLens introduces a new, training-free approach that reimagines deepfake detection as a visual question answering (VQA) task. By harnessing the perceptual power of large vision-language models (LVLMs) and the reasoning capabilities of LLMs like GPT-4, TruthLens not only determines if an image is real or fake — it provides transparent, interpretable explanations for its decisions. Key highlights: ✅ No training required — fully inference-based 🧠 Multimodal reasoning via LVLM + LLM collaboration 🕵️ Evidence-based explanations for detected artifacts 📊 Strong performance on challenging benchmarks This work aims to bridge the gap between detection and understanding, making strides toward more trustworthy and explainable AI systems in the fight against visual misinformation. 👏 Grateful to collaborate with an amazing team: Rajatsubhra Chakraborty, Ali Khaleghi Rahimian, and Thomas MacDougall from University of North Carolina at Charlotte ! 📄 Read the full preprint here: https://lnkd.in/gBgApuc8 #DeepFakeDetection #ExplainableAI #VisionLanguageModels #LLM #MultimodalAI #TruthLens #TrustworthyAI #SyntheticMedia #AIResearch

57

2 Kommentare

Security win: papers and other works submitted/reviewed/published at ACL venues and repositories that discuss novel security issues must now perform industry-standard co-ordinated disclosure. Machine learning and NLP aren't special - we should play by the same safety rules as everyone else. The implementation is such that the disclosure period can run concurrently with anonymous non-public review (as long as a preprint isn't posted during, of course), so this needn't delay researchers in a rush to publish. The documentation of co-ordinated disclosure seems like perfect content for an Ethics section, which doesn't usually count toward page limits, so other paper content doesn't have to be sacrificed as a result of this policy change. Details: https://lnkd.in/drkUDzHq

32

2 Kommentare

Yesterday, at #GoogleIO, we announced Gemini 2.5 Pro #DeepThink, with state-of-the-art performances across math, coding, image understand and much more! Very proud of our research team for the relentless effort over the past 6 months towards advanced deep reasoning with parallel thinking! It's been a wild run over the last 2 weeks to land Deep Thinking just in time for I/O! Heroic efforts across many teams at Google DeepMind from research to production for a mission that felt almost impossible last month! The attached video showed a cool example of how DeepThink can nail a complex solution for an algebra problem at USAMO 2025, something we haven’t seen before in previous models! DeepThink achieved a new state-of-the-art performance of 49.4% (more than double the previous performance) on USAMO 2025, an extremely challenging Math benchmark for AI that requires detailed proof writing. See more details at https://lnkd.in/gEJCNHtJ.

123

1 Kommentar