Category: Technology

Christopher Butler’s take on the best and worst case scenario for AI.

The best case scenario is that AI is just not as valuable as those who invest in it, make it, and sell it believe. This is a classic bubble scenario. We’ll all take a hit when the air is let out, and given the historic concentration of the market compared to previous bubbles, the hit will really hurt. The worst case scenario is that the people with the most money at stake in AI know it’s not what they say it is. If this is true, we get the bubble and fraud with compound motives.

[…]

I don’t worry about the end of work so much as I worry about what comes after — when the infrastructure that powers AIbecomes more valuable than the AI itself, when the people who control that infrastructure hold more sway over policy and resources than elected governments. I know, you can picture me wildly gesticulating at my crazy board of pins and string, but I’m really just following the money and the power to their logical conclusion.

Unmesh Joshi talking about how software engineers discover design during implementation. In my opinion, this is the primary reason using AI in software development is not as straight forward as makers of coding assistants project it to be.

In most mature engineering disciplines, the process is clear: a few experts design the system, and less specialized workers execute the plan. This separation between design and implementation depends on stable, predictable laws of physics and repeatable patterns of construction. Software doesn’t work like that. There are repetitive parts that can be automated, yes, but the very assumption that design can be completed before implementation doesn’t work. In software, design emerges through implementation. We often need to write code before we can even understand the right design. The feedback from code is our primary guide. Much of this cannot be done in isolation. Software creation involves constant interaction—between developers, product owners, users, and other stakeholders—each bringing their own insights. Our processes must reflect this dynamic. The people writing code aren’t just ‘implementers’; they are central to discovering the right design.

Zara Zhang sharing her thoughts on entry level jobs.

A Harvard student told me something I can’t stop thinking about. When they go to the library, every single screen has ChatGPT open. Homework that used to take hours now takes minutes.

But then they talk to alums who say entry-level roles are basically gone. The jobs they planned their entire college trajectory around don’t exist anymore.

AI made homework easier but made proving you deserve a job exponentially harder.

Scary.

There’s a spirited discussion on the research paper A Definition of AGI on Hacker News. This comment by fnordpiglet caught my attention.

Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there – and I’d assert that awareness is all you need.

Erik D. Kennedy reflecting on why has AI not impacted designer, yet.

My hunch: vibe coding is a lot like stock-picking – everyone’s always blabbing about their big wins. Ask what their annual rate of return is above the S&P, and it’s a quieter conversation 🤫

Ha!

An insightful post by Bruce Schneier on the security issues plaguing AI. He also suggests that prompt injections might be unsolvable in today’s LLMs.

The fundamental problem is that AI must compress reality into model-legible forms. In this setting, adversaries can exploit the compression. They don’t have to attack the territory; they can attack the map. Models lack local contextual knowledge. They process symbols, not meaning. A human sees a suspicious URL; an AI sees valid syntax. And that semantic gap becomes a security gap.

Prompt injection might be unsolvable in today’s LLMs. LLMs process token sequences, but no mechanism exists to mark token privileges. Every solution proposed introduces new injection vectors: Delimiter? Attackers include delimiters. Instruction hierarchy? Attackers claim priority. Separate models? Double the attack surface. Security requires boundaries, but LLMs dissolve boundaries. More generally, existing mechanisms to improve models won’t help protect against attack. Fine-tuning preserves backdoors. Reinforcement learning with human feedback adds human preferences without removing model biases. Each training phase compounds prior compromises.

This is Ken Thompson’s “trusting trust” attack all over again. Poisoned states generate poisoned outputs, which poison future states. Try to summarize the conversation history? The summary includes the injection. Clear the cache to remove the poison? Lose all context. Keep the cache for continuity? Keep the contamination. Stateful systems can’t forget attacks, and so memory becomes a liability. Adversaries can craft inputs that corrupt future outputs.

This is the agentic AI security trilemma. Fast, smart, secure; pick any two. Fast and smart—you can’t verify your inputs. Smart and secure—you check everything, slowly, because AI itself can’t be used for this. Secure and fast—you’re stuck with models with intentionally limited capabilities.

The Oatmeal talking about AI art.

As a kid, I had one of those little Casio keyboards where you could hit a button and it’d automatically play a song.

I remember hitting the button.

I remember standing there, pretending to make music.

That’s how I see Al art.

Standing there.

Pretending.

Pretending. This resonated with me. At times I have had this feeling whenever I used AI. While generating proposals. While generating code. While generating ideas. It seems I am pretending to work.

So is the future of work that we all pretend to work?