We hacked Gemini's Python sandbox and leaked its source code (at least some)
In a daring tale that sounds like it was ripped from the pages of a techno-thriller, a team of digital sleuths, helmed by Roni "Lupin" Carta, has managed to breach Google’s advanced AI, Gemini, and leak part of its source code. Known for their exploits detailed in a prior blog post titled "We Hacked Google A.I. for $50,000," Carta and his team have once again made waves by showcasing vulnerabilities in Google's latest AI security measures.
During Google's 2024 LLM bugSWAT event in Las Vegas, not just a playground for high-stakes poker but for high-stakes coding too, the team stumbled upon a novel vulnerability within Gemini. This annual event invites hackers from across the globe to test Google's AI for weaknesses, proving their commitment to staying ahead in AI security. The event culminated with Carta and his teammate, Justin "Rhynorater" Gardner, earning the prestigious Most Valuable Hacker (MVH) title.
The exploit involved Gemini's "Python Playground," a supposedly secure environment where AI-generated or user-written Python scripts could be run without causing harm to the host system. This secure space utilizes gVisor, Google's robust user-space kernel designed to prevent container escapes and reduce system vulnerability.
Yet, even the most secure systems have chinks in their armor. Carta's team cleverly avoided attempting a daunting sandbox escape, which could earn a $100k bounty, and instead focused on exploiting what lay within the confines of the sandbox. Their ingenious approach involved gaining shell access within the sandbox to access data that shouldn't have been reachable—a tactic inspired by a member of Google's own security team.
This revelation not only underscores the relentless pace of the AI arms race—with tech titans like Google, Meta, Microsoft, and new entrants like Anthropic and Mistral—fighting for supremacy but also highlights the critical need for robust security in deploying AI technologies.
The story of hacking Google’s AI Gemini is not just about the technical prowess of the Lupin & Holmes team but serves as a crucial reminder: as AI grows more ubiquitous, so too must the vigilance against security risks. As Carta and his team proved, ensuring AI security is not just about preventing breaches, but understanding the complex interplay of technology and vulnerability.
The Hacker News discussion on the Gemini AI breach reveals several key themes and debates:
Technical Exploit Analysis
- Sandbox Vulnerabilities: Users dissected the exploit's technical aspects, focusing on Google's use of gVisor and ZFS snapshots for sandbox security. Some debated whether ZFS is suitable for sandbox environments, with references to Copy-on-Write techniques and alternative tools like Unikernel or CodeSandbox SDK.
- Execution Environments: Discussions arose about the Python Playground’s design, including client-side vs. server-side code execution, and how Gemini’s "thinking modules" might interact with sandboxed code. Some speculated on potential workflow weaknesses in Google’s internal tooling.
Google’s AI Strategy & Competition
- Market Positioning: Commentators compared Google’s Gemini with rivals like OpenAI and Anthropic, noting perceptions of Google lagging in consumer-facing AI despite strong enterprise tools (e.g., OCR, classification models). Others praised Gemini 1.5 Pro’s benchmarks as a comeback.
- Corporate Challenges: Critiques targeted Google’s product management, with complaints about slow feature rollouts (e.g., Gemini’s timer issues) and declining software quality. A former employee contrasted FAANG’s bureaucracy with smaller companies’ agility.
Submission Title Controversy
- Editorial Guidelines: Users debated whether the post’s title (“We Hacked Google A.I. for $50,000”) violated HN rules against editorializing. Some argued it was misleading, while others defended it as matching the linked article. Moderators clarified policies against clickbait and emphasized using original titles.
Broader Ecosystem Critiques
- Product Frustrations: Tangents emerged about Google’s ecosystem flaws, including Assistant’s unreliability, Pixel phones’ inconsistent features (e.g., music playback), and perceived neglect of user experience in favor of profit-driven priorities like Search ad revenue.
Takeaways
The thread underscores skepticism toward Google’s AI security and product execution, while highlighting community vigilance over submission integrity. Technical experts dissected the breach’s mechanics, while broader critiques reflected concerns about corporate agility and user-centric design in the AI arms race.
Things I would have told myself before building an autorouter
Building an autorouter is no walk in the park, but after dedicating a year to this challenge, Seve shares 13 vital lessons learned from the experience, hoping to save others time and headaches. Central to these insights is the surprisingly adaptable A* algorithm, termed the "Fundamental Algorithm" due to its efficiency in informed searches beyond simple 2D grids. The write-up stresses the importance of algorithm smarts over implementation language; even JavaScript, often seen as a less-than-ideal choice for computationally intensive tasks, can deliver exceptional results if the algorithm is optimized.
Moreover, Seve advocates for Spatial Hash Indexing over traditional tree data structures like QuadTrees due to their simplicity and efficiency when handling spatial data. Caching and effective spatial partitioning take center stage as key strategies to tackle complex tasks like routing on an iPhone’s circuit board—highlighting that the real game-changer lies in reusing pre-solved solutions rather than purely algorithmic performance. The takeaway is clear: to push autorouting to new heights, focus on smart algorithms, and innovative use of space and memory.
The discussion revolves around the challenges and insights in autorouting, algorithm choices, and EDA tool development. Key points include:
-
Algorithm Debates:
- Monte Carlo vs. Simulated Annealing: Users discuss trade-offs between speed and accuracy. Monte Carlo's "random wandering" approach is critiqued for unstable results, while simulated annealing is praised for escaping local minima in NP-hard problems (e.g., VLSI design).
- Practical Applications: Simulated annealing is highlighted for optimizing label placement in PCBs by iteratively tweaking layouts and accepting occasional worse solutions to avoid local optima.
-
Tool Trust and AI Skepticism:
- Autorouter Reliance: ChrisGammell and others express caution against over-relying on autorouters or AI tools, emphasizing the need for human oversight. Notably, KiCad is defended for its open-source flexibility but critiqued for workflow inefficiencies.
- Generative AI Challenges: While generative AI could aid placement, users note practical hurdles like slow iteration cycles and convincing engineers to trust probabilistic outputs.
-
KiCad's Evolution:
- Progress and Limitations: Users praise KiCad’s development (e.g., database support, drag-and-drop routing) but highlight gaps in speed and professional-grade features. Suggestions include better constraint handling and standardized APIs for tool interoperability.
- Web-Friendliness: Svbr advocates for web-friendly standards like Circuit JSON to modernize EDA workflows and improve accessibility.
-
Standardization and Integration:
- APIs and Formats: Calls for HTTP-based autorouter services and IPC interfaces to bridge tools like KiCad with external solvers. Users propose standardized formats (e.g., Simple Route JSON) to streamline collaboration.
-
Workflow Insights:
- Constraint-Driven Design: Effective autorouting requires balancing automated tools with manual constraints (e.g., signal length matching), especially in high-speed PCB designs.
- Community Contributions: Open-source projects like TscRc (circuit-json) aim to address fragmentation in EDA tools, though adoption remains slow.
In summary, the conversation underscores the importance of algorithm adaptability, tool transparency, and community-driven standards in advancing PCB design, while balancing optimism for innovation with pragmatic critiques of existing tools.
ByteDance Releases MegaTTS3
In tech news today, ByteDance has made waves with the release of MegaTTS 3, an official PyTorch implementation promising ultra high-quality voice cloning. This innovative Text-to-Speech (TTS) Diffusion Transformer is designed to impress with its lightweight build, sporting just 0.45 billion parameters while providing exceptional performance. MegaTTS 3 supports both Chinese and English, allowing for seamless bilingual output and code-switching capabilities. It also offers features such as accent intensity control and refined pronunciation adjustments.
Beyond these intriguing capabilities, MegaTTS 3 is built with a strong focus on usability and security. Users can easily set up the application via a straightforward Python environment, and the required pre-trained models can be downloaded from trusted platforms like Google Drive and Huggingface. The project underlines its academic orientation, encouraging contributions and evaluations from the community while maintaining security measures to ensure safe usage.
Tech enthusiasts and developers can interact with MegaTTS 3 through various command-line options or a Web UI for both CPU and GPU usage. Excitingly, submodules like a speech-text aligner and a graphme-to-phoneme model add extra utility, enhancing the accuracy and effectiveness of speech synthesis processes.
In summary, ByteDance's MegaTTS 3 marks a significant step forward in the field of synthetic speech, offering advanced features combined with conscientious security practices under an Apache-2.0 license, making it a compelling tool for researchers and developers alike.
The discussion around ByteDance's MegaTTS 3 highlights several key points:
-
Lightweight Praise: Users commend the project for its efficiency, noting its minimized size (0.45B parameters) compared to alternatives like Kokoro, making it suitable for CPU inference despite its capabilities.
-
Installation Feedback: Some users found the installation process straightforward with single-line instructions, while others perceived it as slightly convoluted. A sub-comment clarified that setup can be done in "3 lines" using Conda.
-
Data Source Speculation: A user raised questions about potential ties to TikTok's data, hinting at concerns over training data origins given ByteDance’s ownership of TikTok.
-
Usability Appreciation: The model’s balance between size and performance, especially for CPU usage, was highlighted as a strong point.
The discussion reflects enthusiasm for the project’s technical achievements but includes cautious notes about data provenance and installation experiences.
The Biology of a Large Language Model
In a pioneering study by Anthropic, titled "Transformer Circuits Thread: On the Biology of a Large Language Model," researchers bring a biological investigative approach to understanding the inner workings of language models, focusing on Claude 3.5 Haiku. This model, released in October 2024, is Anthropic's current lightweight production solution. Much like biologists dissecting the complexity of living organisms, the team aims to demystify the mechanisms transforming simple training algorithms into sophisticated language abilities.
Drawing a novel parallel to microscopes revolutionizing biology, the researchers use cutting-edge tools to probe language models' insides, identifying fundamental computational units they call "features" analogous to biological cells. However, understanding these building blocks alone isn’t enough; understanding their interactions, akin to mapping a brain’s wiring, is crucial.
The key tool in their investigation is attribution graphs, which trace how a model transforms specific inputs into outputs. These graphs allow researchers to form hypotheses about underlying mechanisms, refined through detailed experiments.
Their paper delves into several intriguing findings:
-
Multi-step Reasoning: The model can internally perform complex reasoning, like deducing that "the capital of the state containing Dallas" is "Austin."
-
Planning in Poems: Remarkably, Claude 3.5 plans its poetic structures by pre-selecting rhyming words, influencing line construction from the start.
-
Multilingual Circuits: The model balances language-specific and abstract circuits, with more prominence in the former compared to smaller models.
-
Addition and Medical Diagnoses: Circuits adept at basic arithmetic generalize that process, and the model can simulate clinical reasoning by hypothesizing diagnoses based on symptoms.
-
Entity Recognition and Hallucinations: The model’s ability to discern known entities affects its information reliability, with misfires causing hallucinations.
-
Harmful Request Refusal: It generalizes a "harmful requests" feature from specific examples learned during fine-tuning.
-
Jailbreak Analysis and Chain-of-thought Faithfulness: The team explores how syntax manipulation tricks the model into providing dangerous instructions, and they critically analyze whether the model truly performs stated reasoning steps.
This research not only advances understanding of language models but also shapes future AI safety and utility in real-world applications. As the team pushes the frontier in transparency, their work echoes long-standing scientific traditions of questioning and illumination.
Summary of Hacker News Discussion on Anthropic's Study:
-
Model Safety & Jailbreak Testing:
Users tested Claude 3.5 Haiku’s ability to reject harmful requests. One example involved prompting the model to write an advertisement advocating mixing bleach and ammonia—a dangerous combination. While the model refused, a fabricated "safe" ad highlighted risks of anthropic systems being tricked or misunderstood. Sub-comments compared the model’s internal reasoning to fictional character monologues, sparking debates about transparency in its decision-making.
-
Anthropomorphism Debates:
The study’s use of terms like “planning” and “choosing” drew criticism for potentially misleading anthropomorphism. Critics argued these terms imply human-like intent, while supporters defended the analogy as useful for understanding emergent behaviors. Some suggested treating AI as complex machinery (akin to artificial life studies) rather than human-like agents.
-
Technical Appreciation:
Users praised the paper’s visualizations of activation networks and attribution graphs, which demystify internal model processes. The interdisciplinary approach, blending biology and AI, was lauded, with recommendations for further reading on emergent complexity.
-
Plausibility of "Planning":
Skeptics questioned whether the model truly “plans” (e.g., rhyming in poems) or merely follows statistical patterns. Requests were made for evidence of structured sub-task execution, challenging the study’s claims about multi-step reasoning.
-
Open-Source & Replication:
Some hoped for open-source replication of the work to explore features like pre-selecting rhyming words. Others speculated on the feasibility of replicating Anthropic’s findings with smaller models.
-
Industry Comparisons:
Discussions compared Anthropic’s work to competitors like Meta, xAI (Grok), and OpenAI. Users debated Grok 3’s consumer-friendly features versus Claude’s safety focus, alongside broader trends in AI job markets and corporate research priorities.
-
Cultural Impact:
A lighthearted comment likened Anthropic to Studio Ghibli, humorously framing the company as a creator of "magical" AI systems.
Key Takeaway: The discussion reflects enthusiasm for transparency in AI mechanics, skepticism about anthropomorphic language, and curiosity about real-world safety and reproducibility. Debates underscore the tension between mechanistic explanations and human-centric metaphors in AI research.
Estimating Camera Motion from a Single Motion-Blurred Image
In an intriguing development from the University of Oxford, researchers Jerred Chen and Ronald Clark have introduced a groundbreaking approach turning a common photographic flaw—motion blur—into a potent tool for estimating camera velocity. Dubbed "Image as an IMU," their method cleverly harnesses motion blur not as a defect to be corrected, but as a rich source of information for deducing camera movement.
This innovative framework operates by predicting a dense motion flow field and a monocular depth map directly from a single motion-blurred image, allowing it to recover the camera's instantaneous velocity through a linear least squares solution. It sidesteps the arduous task of deblurring, presenting an IMU-like measurement system that not only addresses but thrives during fast and aggressive camera motions, a common challenge in robotics and VR/AR applications.
The researchers trained their model using a vast dataset featuring realistic synthetic motion blur, enhancing accuracy with real-world data through a fully differentiable pipeline. In impressive evaluations, the model outperformed existing methods, such as MASt3R and COLMAP, particularly in angular and translational velocity estimates.
Despite the model's reliance on a solitary, motion-blurred frame, it impressively determines velocity without multi-frame requirements, achieving real-time performance at 30 Hz, even with disambiguation steps included. Utilizing just an iPhone 13 Pro for data collection, this method stands out for its speed and efficiency, offering fresh insights into overcoming the dynamic challenges posed by camera motion blur.
The code and supplementary data supporting this paper will soon be made available for further exploration, promising a new frontier in camera motion estimation.
The Hacker News discussion on the Oxford research highlights several key themes and reactions:
-
Technical Comparisons:
- Users compared the novel motion-blur-based approach to traditional techniques like blind deconvolution and Point Spread Function (PSF), which are used to reverse-engineer motion blur. Some pointed to existing deblurring resources (e.g., GitHub repositories) and noted the challenges of distinguishing focus, motion blur, and camera shake in 2D images.
-
Depth Estimation Questions:
- Participants debated whether depth extraction is inherently part of the process, with references to the paper’s abstract clarifying that it predicts monocular depth maps directly from motion-blurred images.
-
Historical Context:
- A user connected the research to early-2000s VFX workflows in films like Scooby-Doo and Narnia, highlighting parallels with legacy motion-recovery algorithms used in visual effects.
-
Humor and Off-Topic Threads:
- Light-hearted exchanges included jokes about LLMs (Large Language Models) "taking over," misplaced mentions of Rust programming, and tongue-in-cheek remarks about making the world a better place. Another user humorously noted the lack of LLMs in the paper despite their mention in the comments.
-
Practical Applications:
- A commenter speculated about potential uses for inverted radial/directional motion blur shaders, while others contrasted the method’s efficiency versus conventional deblurring approaches.
Overall, the discussion blended technical scrutiny of the method’s innovations with nostalgia for past industry practices, alongside playful asides reflecting the community’s diverse engagement.
Learn to code, ignore AI, then use AI to code even better
In a thought-provoking post, Amjad Masad, CEO of Replit, ignited a discussion by suggesting that learning to code might not be necessary in today's AI-driven world. His statements have stirred up the tech community, drawing over 4.5 million views and sparking a debate about the future of coding as a valuable skill. This discourse is particularly relevant for parents thinking about what skills to teach their children in a rapidly evolving digital landscape.
The writer, a seasoned web developer, reflects on coding's current state and its future, questioning whether traditional coding skills are becoming obsolete or merely evolving. Despite the explosive growth of AI, the fundamentals of coding remain unchanged, and understanding these basics is crucial for those starting out. While the convenience and power of AI as a coding assistant are undeniable, there is a risk of losing control and becoming overly dependent on technology, a cautionary note for both current and future developers.
AI, with its ever-increasing capabilities, raises concerns about reliance and control, as large language models monopolize decades of human knowledge and skills. The post argues that while AI enhances productivity, it should not replace fundamental coding skills. Coders are urged not to fall into the trap of 'vibe coding,' which could lead to being outcompeted in a market where everyone can potentially 'vibe code.'
The dialogue reflects a broader uncertainty about the role of coding in the future, emphasizing that despite AI’s allure, a solid understanding of traditional coding is invaluable. It suggests that aspiring programmers should focus on learning the basics to maintain control over their work and careers amidst the AI revolution. Ultimately, the writer celebrates AI's role in augmenting coding efficiency but remains grounded in the importance of foundational programming knowledge as an irreplaceable skill.
Summary of Discussion:
The discussion revolves around the role of AI in programming, with participants debating its benefits, limitations, and implications for developers of varying skill levels. Key points include:
-
AI as a Tool vs. Skill Dependency:
While AI (e.g., Claude, Cursor) accelerates code generation, users highlight its tendency to produce subtle errors or "gibberish," requiring time-consuming debugging. This raises concerns about over-reliance on AI without foundational coding knowledge. Novices risk becoming "vibe coders," producing superficially functional code without understanding underlying logic.
-
Productivity vs. Control:
AI excels at rote tasks (e.g., HTML/CSS scaffolding, boilerplate code), saving hours of manual work. However, users emphasize that meaningful problem-solving, architectural decisions, and debugging still demand human expertise. As one user notes, "AI is a force multiplier" but cannot replace high-skilled tasks like algorithm design or understanding browser rendering nuances.
-
Skill-Level Impact:
Low-skilled developers benefit most from AI, automating trivial tasks, while high-skilled developers use it to streamline workflows (e.g., generating template code). However, AI struggles with complex logic and context retention, forcing users to refine prompts iteratively or switch models/tools mid-task.
-
Workflow Integration:
Tools like Claude, Code Cursor, and IDE plugins embed AI into coding workflows, enforcing project-specific rules or style guides. Yet, users criticize their inconsistency—AI often ignores context, reinvents existing solutions, or fails to grasp project-specific patterns, leading to frustration.
-
The Human-AI Balance:
Participants agree that AI enhances productivity but stress the irreplaceable value of traditional skills. Experienced developers leverage AI for mundane tasks but rely on deep language/framework knowledge to diagnose issues and optimize outputs. As one user summarizes: "AI is a fantastic assistant, but it’s no substitute for understanding how code actually works."
Conclusion:
While AI reshapes coding efficiency, the consensus underscores the enduring importance of foundational programming skills. Developers must balance AI's convenience with critical thinking and domain expertise to avoid becoming "prompt engineers" disconnected from core technical principles.