Category: Artificial Intelligence

Robert Greiner reflecting on the recent acquisition of Bun by Anthropic.

Leaders don’t express their true beliefs in blog posts or conference quotes. They express them in hiring plans, acquisition targets, and compensation bands. If you want to understand what AI companies actually believe about engineering, follow the cap table, not the keynote.

This intriguing comparison about cars and AI which highlights how we have built our cities around cars. While cars pollute and take up a lot of space in parking, they are incredibly useful for humans.

This comment on Hacker News by rukuu001.

I think a lot about how much we altered our environment to suit cars. They’re not a perfect solution to transport, but they’ve been so useful we’ve built tons more road to accommodate them.

So, while I don’t think AGI will happen any time soon, I wonder what ‘roads’ we’ll build to squeeze the most out of our current AI. Probably tons of power generation.

This comment on Hacker News by sotix.

This is a really interesting observation! Cars don’t have to dominate our city design, and yet they do in many places. In the USA, you basically only have NYC and a few less convenient cities to avoid a city designed for cars. Society has largely been reshaped with the assumption that cars will be used whether or not you’d like to use one.

What would that look like for navigating life without AI? Living in a community similar to the Amish or Hasidic Jews that don’t integrate technology in their lives as much as the average person does? That’s a much more extreme lifestyle change than moving to NYC to get away from cars.

Charlie Warzel talking about how generative AI’s perpetual beta has put us all in perpetual uncertainty.

The world that ChatGPT built is a world defined by a particular type of precarity. It is a world that is perpetually waiting for a shoe to drop. Young generations feel this instability acutely as they prepare to graduate into a workforce about which they are cautioned that there may be no predictable path to a career. Older generations, too, are told that the future might be unrecognizable, that the marketable skills they’ve honed may not be relevant. Investors are waiting too, dumping unfathomable amounts of capital into AI companies, data centers, and the physical infrastructure that they believe is necessary to bring about this arrival. It is, we’re told, a race—a geopolitical one, but also a race against the market, a bubble, a circular movement of money and byzantine financial instruments and debt investment that could tank the economy. The AI boosters are waiting. They’ve created detailed timelines for this arrival. Then the timelines shift.

We are waiting because a defining feature of generative AI, according to its true believers, is that it is never in its final form. Like ChatGPT before its release, every model in some way is also a “low-key research preview”—a proof of concept for what’s really possible. You think the models are good now? Ha! Just wait. Depending on your views, this is trademark showmanship, a truism of innovation, a hostage situation, or a long con. Where you fall on this rapture-to-bullshit continuum likely tracks with how optimistic you are for the future. But you are waiting nonetheless—for a bubble to burst, for a genie to arrive with a plan to print money, for a bailout, for Judgment Day. In that way, generative AI is a faith-based technology.

It doesn’t matter that the technology is already useful to many, that it can code and write marketing copy and complete basic research tasks. Because Silicon Valley is not selling useful; it’s selling transformation—with all the grand promises, return on investment, genuine risk, and collateral damage that entails. And even if you aren’t buying it, three years out, you’re definitely feeling it.

Sami Bahri emphasising that while AI accelerates the process, the intelligence to setup and monitor that process still requires a human.

Intelligence implies wisdom, context, and nuance. While AI models are simulating reasoning better every day, in a business context, they are fundamentally pattern-matching engines. They excel at acceleration.

• The Old Way: An analyst reads 50 contracts (unstructured), highlights risks based on gut feeling (unstructured process), and summarizes them in 3 days.
• The AI Way: An AI scans 50 contracts and extracts specific risk clauses based on defined parameters in 3 minutes.

The process (Review Contracts -> Identify Risk -> Summarize) hasn’t changed, but it had to be rigorously defined for the AI to work. The intelligence (knowing what a “risk” actually means) still requires human governance. What has changed is the velocity.

Christopher Butler’s take on the best and worst case scenario for AI.

The best case scenario is that AI is just not as valuable as those who invest in it, make it, and sell it believe. This is a classic bubble scenario. We’ll all take a hit when the air is let out, and given the historic concentration of the market compared to previous bubbles, the hit will really hurt. The worst case scenario is that the people with the most money at stake in AI know it’s not what they say it is. If this is true, we get the bubble and fraud with compound motives.

[…]

I don’t worry about the end of work so much as I worry about what comes after — when the infrastructure that powers AIbecomes more valuable than the AI itself, when the people who control that infrastructure hold more sway over policy and resources than elected governments. I know, you can picture me wildly gesticulating at my crazy board of pins and string, but I’m really just following the money and the power to their logical conclusion.

Unmesh Joshi talking about how software engineers discover design during implementation. In my opinion, this is the primary reason using AI in software development is not as straight forward as makers of coding assistants project it to be.

In most mature engineering disciplines, the process is clear: a few experts design the system, and less specialized workers execute the plan. This separation between design and implementation depends on stable, predictable laws of physics and repeatable patterns of construction. Software doesn’t work like that. There are repetitive parts that can be automated, yes, but the very assumption that design can be completed before implementation doesn’t work. In software, design emerges through implementation. We often need to write code before we can even understand the right design. The feedback from code is our primary guide. Much of this cannot be done in isolation. Software creation involves constant interaction—between developers, product owners, users, and other stakeholders—each bringing their own insights. Our processes must reflect this dynamic. The people writing code aren’t just ‘implementers’; they are central to discovering the right design.

Zara Zhang sharing her thoughts on entry level jobs.

A Harvard student told me something I can’t stop thinking about. When they go to the library, every single screen has ChatGPT open. Homework that used to take hours now takes minutes.

But then they talk to alums who say entry-level roles are basically gone. The jobs they planned their entire college trajectory around don’t exist anymore.

AI made homework easier but made proving you deserve a job exponentially harder.

Scary.

There’s a spirited discussion on the research paper A Definition of AGI on Hacker News. This comment by fnordpiglet caught my attention.

Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there – and I’d assert that awareness is all you need.

Erik D. Kennedy reflecting on why has AI not impacted designer, yet.

My hunch: vibe coding is a lot like stock-picking – everyone’s always blabbing about their big wins. Ask what their annual rate of return is above the S&P, and it’s a quieter conversation 🤫

Ha!

An insightful post by Bruce Schneier on the security issues plaguing AI. He also suggests that prompt injections might be unsolvable in today’s LLMs.

The fundamental problem is that AI must compress reality into model-legible forms. In this setting, adversaries can exploit the compression. They don’t have to attack the territory; they can attack the map. Models lack local contextual knowledge. They process symbols, not meaning. A human sees a suspicious URL; an AI sees valid syntax. And that semantic gap becomes a security gap.

Prompt injection might be unsolvable in today’s LLMs. LLMs process token sequences, but no mechanism exists to mark token privileges. Every solution proposed introduces new injection vectors: Delimiter? Attackers include delimiters. Instruction hierarchy? Attackers claim priority. Separate models? Double the attack surface. Security requires boundaries, but LLMs dissolve boundaries. More generally, existing mechanisms to improve models won’t help protect against attack. Fine-tuning preserves backdoors. Reinforcement learning with human feedback adds human preferences without removing model biases. Each training phase compounds prior compromises.

This is Ken Thompson’s “trusting trust” attack all over again. Poisoned states generate poisoned outputs, which poison future states. Try to summarize the conversation history? The summary includes the injection. Clear the cache to remove the poison? Lose all context. Keep the cache for continuity? Keep the contamination. Stateful systems can’t forget attacks, and so memory becomes a liability. Adversaries can craft inputs that corrupt future outputs.

This is the agentic AI security trilemma. Fast, smart, secure; pick any two. Fast and smart—you can’t verify your inputs. Smart and secure—you check everything, slowly, because AI itself can’t be used for this. Secure and fast—you’re stuck with models with intentionally limited capabilities.