Engineering and unpredictability aren’t often thought of as compatible. When I think of engineering I think of precision, accuracy, reliability… I think of bridges that stay up, planes that take off and fly, nuclear plants that provide power… Ok, I’ve also seen each of those things fail in my lifetime. But mostly, they fulfill their promise. The promise to take you across this river, this ocean or to provide electricity for an entire city. Those are important promises to fulfill!
Software Engineering gets a little more slack to some extent. Generally speaking, people’s lives aren’t threatened by dysfunctioning software. A website goes down, an app crashes, a feature is buggy… It's annoying. But life goes on. Of course I could list a thousand exceptions to that, but you get the idea.
So what’s interesting to me here is the level of inherent unpredictability that you sign up for when you build AI powered tools. After doing that pretty intensely for the past 6 weeks, I can definitely say that there is still a prominent “black-box” feeling about them. So when the whole premise of your project rests on needing AI to act in a predictable way, you know you’re in trouble.
There is something absurd about wanting a prediction machine to be… predictable. Isn’t it? And yet that’s pretty much what LLMs are, particularly text to text models like ChatGPT and Claude: they predict what we want to hear. This is an oversimplification of course, if you want to go deeper, perhaps look up Andej Karpathy’s videos. But the point is, these “machines” are non-deterministic. Give them the same prompt over and over again and chances are you’ll get a different result every time.
So how do we work with that? Well right now, the only way we have to influence the outcome is through dabbling in the arcane art of “prompt engineering”. It’s crazy to think that’s all we got: words. No knobs, no sliders, no handles. Just words and a lot of hope.
“Do this like that”, “don’t do this”, “here is an example of a good answer”, “here is a bad answer”, “CRITICAL: DO NOT FORGET X”. “You are a software engineer instructor…”, “You are a movie Genie who knows about the entire history of cinema”, “You are a gobelin in an enchanted forest…”.
And yet, despite my best attempts, I know I can never be sure of what will come out of the AI ether. But text models at least give you something to iterate on - you can refine, clarify, build on previous responses.
Image models? That's a whole other level of chaos.
I have very limited experience with text to image models and so far it’s been a mix of wonder, laughter and frustration. More or less in that order.
Usually these tools have brought me somewhere between 50-75% of the way towards what I want. That’s pretty good. Except, unlike with coding and words, I can’t edit the damn thing. I’m left having to try to iterate with more words. And as I came to find out, text to image models work in a way that makes them pretty clunky at iterating. They’re essentially generating a new image from scratch every time you run them. There’s hardly any sense of continuity. It’s extremely frustrating. “I just need the lighting to be less orange and the look to be more photorealistic”. Boom. Something that has nothing to do with the previous image. A simple aspect ratio change? A completely different image. “More bubbles in the tank please”? We’re in an entirely different world. Three hours later, I felt like I had gambled my credits away. Every time I’d send a prompt, I felt like I was putting a coin in a slot machine. Not only do I not particularly enjoy gambling, I also find it really inappropriate at work!
It feels like there's a desperate need for a middle ground between a software like good old Photoshop, with its surgical precision, and the wild, generative power of AI tools like Flora. Something more nuanced. A tool that lets you choose your instrument, whether it's a chisel, a chainsaw, or anything in between, and shape your creation with both intention and freedom.
I would absolutely love to use a tool like that. If you’re building one, I want to work with you.