Welcome to the Maintainable Software Podcast, where we explore the art of improving existing

software with seasoned practitioners who have helped organizations overcome the problems

often associated with technical debt and legacy code. I'm your host, Robbie Russell. On this

episode, Obi Fernandez, who is a software engineer, speaker, DJ, author, and is currently

chief scientist at Olympia. Obi joins us from Mexico City, Mexico.

Obi Fernandez, we're so glad to have you join us on Maintainable. Welcome.

Thanks, Robbie. Glad to be here.

So as you reflect on your experience in the industry, what do you believe are a few common

characteristics of, dare I say, well-maintained software?

Oh man, we're just getting right into it. I mean, definitely having some test coverage,

because if you have test coverage, you can make changes. But how boring must that answer

be? I'm guessing every guest says that.

Some variation of that.

But one of the reasons after nearly two decades that I still love Ruby and still insist on

using it, even these days where almost everything that I'm personally involved in is done in

Python, it seems like. I think Ruby syntax leads itself, especially for someone who's

been doing it for so long since me, you know, like me, is to write readable code. And I

think that's a very important factor in your software being maintainable is, you know,

can I go back to what Obi from six months ago or Obi from a year and a half ago wrote

and figure it out? And I think my track record's pretty good on that. And I think Ruby is to

blame for some of that in terms of writing code that is fluent and, you know, reads like

natural language. And yeah, I mean, that's my answer.

I think about also you mentioned test coverage and precursor to this conversation, you had

also mentioned that code that's like robust code. And I was kind of curious, maybe hear

your thought on like, what makes code robust? Like, how would you kind of delineate that?

Well, I think if you've been doing this long enough, we're all familiar with like the code

that blows up for no, for no explicable reason. I've gotten a lot of that in the last year,

the app that I'm currently working on is called Olympia. And it, it probably is the most post

modern software I've ever written. I've hit a new peak of like external dependencies with,

you know, all the different APIs and kind of offloading major subsystems to providers and

things. And even the main one, which is OpenAI, which I use for, you know, for GPT-4 for the main

conversation loop of Olympia. It's the engine behind how our virtual assistants communicate

with our users. And over the last six months, I think I've done a pretty good job of making

the conversation loop robust in the sense that pretty much anything that OpenAI throws at me in

terms of error messages, I can handle gracefully, you know, whether it's retrying or giving feedback

to the user about what's going on or alerting customer support that they can be on top of it.

But it's taken a while to get there. There's definitely been an investment, you know,

in the less mature that the services that you're integrating with, the more uncertainty you tend

to have in terms of what's going on. Like I can think off the top of my head of like,

probably at least six or seven different ways that a request to OpenAI can fail,

you know, at the moment. That's interesting. You know, I was thinking about, you know,

definitely we'll get into talking about your book and stuff like that and definitely talk a little

bit about the software project you're primarily focused on right now, Olympia. But when you're

integrating with external dependencies, APIs on any software project can be a little challenging.

But I don't know that I've, because I personally haven't done much integration at all with any of

like, one, it's trying to find good useful use cases for particular projects we're already

working on. And it hasn't just been necessarily a big requirement for most of our current client

base. But when I try to wrap my head around, like how do you validate if it's a good response or not

from something that's, it's not like a data point that you can easily just check, like this thing is

a hundred percent accurate data point. And this, you know, how do you think about that when you're

having a conversational based back and forth with an API and stuff like that, and it gives you

something that's relatively confident in itself, but is this actually a match that you're saying,

pulling back weird areas, but like, it's like,

Oh man. So, so I would, I wasn't even talking about the, I wasn't even talking about like the

qualitative, you know, kind of more subjective, you know, correctness of the response. I was

talking about whether the response even worked to begin with. I've had to familiarize myself

with a whole gamut of, of HTTP error codes that I was, you know, kind of had forgotten for a while,

like 426, 427, you know, different kinds of five hundreds. Cause we get them all. Cause I mean,

there's, I'm picking on OpenAI, but it's just because, you know, we, we have a, they have a

fair amount of instability in their API. I think sometimes they're just, they just have too much

load instability, which is not reflected on their status page. So, I mean, you can't even just

necessarily rely on that to know whether something's going on. Luckily for the most part,

if you have a smart retry strategy, you just end up waiting. You do a little bit of back off and

it's usually okay. The other thing that's nice is that there's, at least for now, the users have a

super high tolerance of errors because chat2BT is just, isn't reliable itself and you know,

will fail. It's, it's really an interesting topic. It's a different topic, but it's an

interesting topic in terms of eval. Evals is what you call, you know, kind of systems or patterns

that are meant to, to check the correctness of the response, whether it aligns with what your,

you know, particular guidelines are. So there's different kinds of evals that you can do. And

actually this is something I get to in the book on the final chapter called quality control.

So there's kind of two sides of the same, two faces of the same coin. There's evals and guardrails

and guardrails are what you implement to eval on the fly the responses you're getting back from the,

from the AI. So for instance, a really simple guardrail that we have is to make

sure that only official languages are, the AI only responds in official languages.

And we do that with a super, super cheap model. That's really, really fast. Essentially we just

want, we want to spot check and make sure that no one's like kind of hacked our system to talking

Vietnamese or Chinese or something that blows up our costs by like 10x because of the nature of

Hey folks, it's me, Robbie. I want you to take a moment, close your eyes. Now picture your code

and your applications as a symphony. Now to keep that symphony playing smoothly,

you need an orchestra of tools. That's where our podcast sponsor AppSignal takes center stage.

They combine the elegance of error tracking, the precision of performance monitoring,

and the harmony of logging into one symphonic suite. Whether you're composing with Ruby,

jamming with Elixir, orchestrating with Node.js, or harmonizing with Python,

or maybe even a little bit of flourish of JavaScript, AppSignal's got the sheet music

for you. And here's the crescendo. Plans start at just $23 US a month. That's got to be music

to your budget's ears. Plus they're certified ISO 27001, and they dance the GDPR and HIPAA

compliance beats. So don't miss a beat, my friends. Head on over to AppSignal.com

meet your good friend, Robby from Maintainable Centia. Now open your eyes and let the symphony

of smooth coding begin. Let's get back to our show. So you could potentially, it could be very

easy for someone if they just loosely integrate with something like OpenAI to kind of change

how they're using it, whether it's malicious or not, or they're just changing the language and

that adds a whole nother level of complexity to the infrastructure. For this kind of app,

it's a conversational app and you're letting the user interface directly with the AI.

Our users are great. They pay quite a bit to use the service. So we haven't really had too many

out and out flagrant abusive cases or things where people were wasting resources or using us

inappropriately. A lot of the things that we implement that are kind of like extra layered

on top, whether it's guardrails or eval, kind of after the fact based evaluation of the conversations

for quality control, for customer satisfaction reasons, things like that. They're just that,

I mean, they're, you know, they're meant to provide a better service, right? But AI changes

the game because it's no longer just about the quality of your code, you know, that makes things

maintainable. I mean, some parts of what, some of the ways that you can use AI, and this is probably

what's most unique about my book and what I'm most excited about and what readers seem to be

most excited about as well is that I advocate for dropping in little AI black boxes as discrete AI

components to replace otherwise traditional implementations of functionality. So for

instance, I've been playing around with this concept within Olympia for some of our account

management. So in Olympia, you subscribe to one of three base plans that gives you a team of AI

powered consultants. And starting about four or five months ago, we started adding the ability

to add extra assistance to your plan. You know, if you're an experienced, if you're listening and

you're an experienced software developer, you can imagine, okay, there's like kind of a catalog of

extra, you know, extra bots you can add to your account. So you look at those and you have a

button that lets you add them. And then there's some validation that goes on. And then I have

to create a new line item and add it to their Stripe subscription.

And I have to modify them as, you know, it's kind of like the typical things you would expect when

account teams request. So when I was implementing that, instead of doing it in what I would call

just a normal traditional kind of way, what I did was I modeled an account manager class, almost

like a microservice, the way you think about a microservice, that's responsible for that kind of

change. And I made like a little worker. In the book, I call this approach multitude of workers.

So I made a worker who is my account manager. And the interface between the rest of the app

and this account manager is English, it's plain text. So basically, if the user goes in and looks

at an extra assistant they want to add to their account, and they push that button to do so,

internally, there's an English statement that's generated, you know, please add bot config

with a UID to account with a UID. That's packaged inside of an account change request. And it's

sent to this worker. And then the worker looks at the request. And it has at its disposal,

a number of tool functions that allow it to do a number of things, including adding new bot

configs to an account. But it can also send an email to the user or to customer service or both.

It can also reference Olympia documentation and guidelines. And it can also finalize the request.

So then at that point, programming it and maintaining that business logic, let's say,

around how that account manager works becomes a matter of prompting. So the prompt that's given

to account manager tells it how to behave. And in fact, while we're talking, I'll fire up my editor

and I'll bring it up so I can actually read it. But the instructions literally tell it what to do

when it gets a request. I'll read you the system directive, quote, unquote. This is the

configuration for this bot, right? You are an account manager for Olympia. The user will request

changes to their account and you will process those changes by invoking one or more of the

functions provided. Make sure to notify the account owner of the result of the change request.

Always end by calling the finished function so that we save the state of the change request

as completed. That's it. Between that and the functions that it gets and the functions it has

is add bot config to account, add new line item to subscription, notify account owner, and completed.

Those are the functions that it has available to it. Those are just normal methods that have

some imperative logic associated with it. Add a bot config to an account basically starts a

transaction, finds the right bot, finds the right account, adds it to it. And then the output of

those functions give a little bit more instructions as well. So for instance, if you add a bot config

to the account, the output of that function, as opposed to being, I don't know what you would

typically return from a function like that, but the output of this particular function is bot

config with name, fill in the blank, add it to user's account, make sure to use the bot's name

when communicating with the user. And then it says new account, new client for bot config name,

account name signed up. I'm kind of going through this a little quickly. Make sure to add a line

item to the user's subscription. The price is fill in the blank. The monthly charge in cents is

fill in the blank, right? So that's what's returned to the bot when it calls that function.

The reason I call it multitude of workers is because it changes your perspective on how

implementing logic in your system to being more like you have a little human that is following

instructions and you plop them into your code base. And then in kind of thinking through that,

do you feel like that approach is just helping you kind of think about how you could potentially

tap external workers or things like that versus how you're doing it within your own internal app?

Because you could be like, click a button and it upgrades you through the normal process, but for

you to change your mental model of how it's its own bot in a way that you're interacting with,

you know? And so then it's taking advantage of that in that sort of way. Then is it safe

assume that when the emails might come from the account manager or whoever the emails get sent,

they might be a little bit more tailored toward the particular... They might not always be the

same because it's not going to be using the same email. Your account's been upgraded very

consistently. Everybody gets the same email or might have a little bit more personalization and

flavor and feel like you're interacting with an application, but you're actually interacting with

an organization, with a team of people that you're working with in some capacity. There's

air quoting, account manager over there that's helping take care of me, make me feel like a

little bit more of a white glove service. I could have it do that, right? It would probably be as

easy as just saying, insert a little bit of friendliness into it. I didn't in this case.

it for free is that if something goes wrong or something is, uh, is different

than what is expected, it is more robust in the sense that it can, it can deal

with it, uh, or it can escalate to a human in the loop to fix it, but kind

of going back to something you said a minute or two ago, it does allow me to,

to do some pretty cool things that, that I wouldn't be able to do as

easily with traditional code.

And I'll give you a concrete example.

We rolled this out and very soon thereafter, I got a Slack message from

our customer service team saying, Hey, we had a failure with this new code.

Uh, a user with an account that was inactive because they had

scheduled it for cancellation, added a bot.

And we don't think that that should be allowed, like you

should have an active account.

And I said, I think you're right.

Like you shouldn't be able to add an account.

It causes operational issues.

Right?

So JP said, you know, do you want me to add a story for that?

And I said, no, I think I got it because I literally just opened up this

file I have in front of my face right now.

And I added a line to the directive that I didn't read the first time through.

So after it said, make sure to notify the account owner of the

result of the change request.

I inserted one line that said, do not allow the user to add a new bot config

unless their account is active as part of what, uh, is provided to this bot.

They get a dump of the account.

So they, this bot knows whether the account is active.

So with the addition of that one statement into the directive, I was

able to handle the change request, which was don't let people with

inactive accounts, add bots.

Right.

Not only that, if the, and I tested this to make sure if you do try to add a

bot and you have an inactive account, the bot will write to you and say, Hey, I know

you requested this to happen, but your account is inactive.

If you'd like to try again after making your account active, you know, please do so.

It's kind of blowing my mind a little bit and in a good way.

Uh, trying to, you know, thinking about this, you know, I'm curious, uh, take

quick step back cause you've mentioned your book a few times and I'll mention

it in the intro section there, but could you share like a little bit about

inspired you to write your upcoming book, uh, patterns of application

development using AI?

Yeah, sure.

I mean, I, um, like over a year ago when I started working on Olympia, because

I've written books before, and this is such a greenfield, you know, with AI and

stuff, I started taking notes of things that I thought were novel or, you know,

cool in different ways for any of you out there who dream of writing a book.

That's, that's one of the main things you should do, you know, like use

Evernote or whatever, you know, and just keep track of things.

I think Kent Beck or maybe it was Martin Fowler used to use the actual index

cards and told us about that.

So, you know, eventually what you do is when you have a stack of those index

cards, whether they're physical or virtual, you know, then you have a book

because you have content to write about.

It's very hard to sit down and just like materialize a book out of nowhere.

Right.

So I've been doing that for a while.

And then in Vegas, my old friend, Tom Rossi raised his hand during my, a talk

I was giving about AI, which you may have been in the room.

He said something like, Oh, this, this all seems really cool what you're

talking about, but I'm finding this AI overwhelming, like, how do I get started?

You know, how would you recommend I get started?

And that's a really good question.

And I was like, Oh, I had a voice in the back of my head going, there's

got to be like a million people asking this same exact question right now.

You know, AI is this big, big thing.

That's kind of like encroaching on all of our lives, whether we want it to or not.

And so what do you, what do you do with it?

Like, how do you get started?

If you do as much as just subscribe to one or two AI newsletters, it's

already overwhelming because literally every, literally every day you're, you

can have an avalanche of new things going on and whatnot.

So I said, okay, cool.

Let's write a book that describes how to get started.

So I took inspiration from classic Greenfield books, like the Gang of

Four Patterns book, Patterns of Enterprise Application Architecture

Martin Fowler, Patterns of Enterprise Integration, you know, by, by Gregor

Hopi and David Rice, old colleagues at ThoughtWorks from when I worked there

in the 2000s, and I said, I want to write a book like that, like that is

literally, you know, has a shot at becoming like the reference book for how

people talk about integrating AI components into their application development.

And I took a very, very specific approach, which was to ignore

things that I feel are transient.

So like the whole debate about what AI's appropriate role is, whether it will

replace engineers, specific technologies that are current, you know, currently

in vogue or whatnot, like I tried to, I tried to stay away from those

topics and not cover those.

And I tried to take real code from Olympia and show how they represent

patterns of interaction between an AI and a normal code base, and, and very

much putting the emphasis on normal day-to-day application development

that everyone does.

So the examples are in Ruby on Rails, because that's what I know.

I extracted the code examples in the book from Olympia.

They're meant to be very practical.

There's no like car and wheels, like you might see in a, like an OO primer

or, you know, stuff like that.

It's all kind of real examples, you know, of things.

And then in terms of what the book covers, part one starts from the beginning,

like kind of if you, you know, going through a primer of what is a large

language model, how does it work?

What are the components of it?

What is a late, you know, what is the latent space?

I mean, I talk about narrowing the path for when you're trying

generate something with, with an AI.

And why do I call it that?

You know, like, because these frontier models, which is the terminology

they use to refer to GPT-4 and Cloud3, like the really, really big ones.

For intents and purposes, they know everything that there is to

know at at least a low level.

And because of that, you know, take any three letter acronym that you throw

at it, just to give you an example, and there's probably 20 different

versions of it that it could, that you could be talking about, so they get

confused, you know, like if you don't narrow the scope of what you're asking

for and what you're talking about, and that that's an art, you know, at this

point, there's a lot of craft in using these things, but you can learn it, you

know, and that's what people refer to as prompt engineering.

So there's a lot of material that's just kind of fundamental material

about prompt engineering in there.

And how, you know, how do you come up with a prompt?

How do you refine prompts?

How do you manage them?

How do you version them?

How do you template them?

You know, things like that.

So there's a lot of that kind of fundamental stuff.

If you're going to use AI to drive intelligent workflows, how do you think

about that, you know, from an architectural perspective, how do you model it?

In your mind, how do you model it in terms of how you will implement it?

If you're going to use AI to do really cool things like self-healing data or

intelligent error handling, you know, when we talk about robustness, I mean,

you can use AI to do pretty cool things for your application in terms of robustness.

So the example I give the book in terms of self-healing data, I mean, we, we

deal with a lot of JSON because we have, we have the AI models generate JSON.

And because they are not perfect at generating anything, really, they make

mistakes, just like, like if you were a human writing JSON, you would make,

you would make mistakes, right?

You would forget quotes, you would mess up your quotation or things like that.

Or you would deviate from schemas and they do the same thing.

So you go through enough of these errors and you have it blow up your code enough.

And you start thinking, maybe I can just get it to fix the JSON that was broken.

Just give it another crack.

So you can do stuff like that.

It's not so much a retry, but like, if you have broke, you know, like if you have

broken JSON and you want to try to get some value out of it, out of it, like you

can have, we have a JSON fixer class.

So you give it the JSON and you give it the error message and you say, Hey, here,

fix this while preserving what it, what it's supposed to do or what it meant.

We'll be back with our interview with Obi in just a moment.

Hi, it's me, Robbie.

I just want to take a quick moment to say thank you for listening to Maintainable.

If you're finding these discussions valuable, please consider sharing them

link amongst your peers and or writing a review on Apple podcast or a

Spotify to help spread the word.

Also, do you think there's someone I should have on the podcast sometime?

Shoot me an email to Robbie with a Y at maintainable.fm and state your case.

And now let's get back to our interview, Obi Fernandez.

You know, I think about, you know, like when you're building out applications

and taking advantage of using these types of, you know, these LLMs and

their, their API endpoints that they're offering us, I'm also curious if you've

thought much about how just maintenance in the work of the projects that you're

working on, how, how AI can be a benefit to just, you mentioned self-healing as

a thing, but also just from like coming to existing legacy code.

I know that you primarily, maybe these days work a lot more greenfield

applications, but I think about projects that are around for awhile and like, how

are you thinking about how AI may or may not be able to aid in the development

process and help maybe expedite some of those, the challenges that some teams

face from accruing a lot of technical debt or just being really outdated on

versions and stuff like that and dealing with upgrades and removing some of that

fear, knowing that AI is not perfect, but neither are humans.

I can't really speak to that.

I don't have direct experience with it to the extent that I've used, like I tried

using a new product from GitHub called, I think called code workspaces or something

like that, they have a beta, there's a waiting list.

I got, I finally got access to it.

And I don't know if it's just bad with Ruby, it's entirely possible, but the way

that this thing works is it supposedly has access to your whole code that you

have, you know, when you add a repo to it and then you, you start a new session, I

believe it's what it's called.

And then you add a, you know, you, you say what you're working on.

So it's like changing a feature, fixing a bug or whatever.

And then it goes through and it creates a plan of action and it runs it by you.

And if you agree with the plan of action, I mean, you can modify it if you want, but

it, you know, once, once it's good, you hit go, and then it uses AI to kind

of do the work for you.

So I tried to do that with Olympia and to have it, so, so I, I, you're working

with these frontier technologies, you know, you end up, you know, things

change out from under you.

So like, for instance, in Olympia, when I started out, I built it on MagnetChat,

which is an open source chat to BT clone that I wrote and like back in February

of last year, and so some things were still using kind of the magma chat

backend to do requests to the AI.

And nowadays, most of my stuff is switched over to Ray, which is my library

for, for doing AI that I talk about in the book.

So I took one particular class that was still using magma.

And I said, I want to switch this over to use this module chat

completion, which is from Ray.

And I was pretty excited about it.

And I hit go, and then it thought about it and it's even animated.

So it showed me that it was doing stuff.

And then it just deleted like 80% of the classes that I was done.

And what it left, what it left me with was nonsensical, just didn't work.

So I think it's quite not, not quite there yet, at least not for Ruby or at

least not for big, you know, big apps, which Olympia at this point is, so.

Yeah, it's interesting.

Yeah.

I'm always kind of curious about it on that front, thinking about, will there

be some, be able to expedite some of those painful things that some teams are

dealing with and I think it might, might still be a little bit out there for us.

But I'm also curious a little bit more about, can you tell us a

little bit about Olympia as well?

You kind of hinted at it a little bit, but like what's, what's the

core idea behind the product?

The core idea behind the product is to provide as close as we can get with

current technology to a human team mate for your small business.

So we have like 12 or I think we have 14 now different kinds of profiles.

So it's like things like a legal consultant, business strategist, social

media expert, content specialist, copywriter, and some of them have special

tools for, so like for instance, the copywriter understands how to create a,

to collaboratively create a content brief with you.

Let's say you want to write a blog post or something.

So it will go over and you define the requirements and then it will, it has

tools for writing long form content in one shot, so it can generate like

thousands of words, you know, kind of complicated outlines from a brief

compared to what you get out of a chat TPT, which, you know, you can, you

can tell chat TPT, I want to write a blog post and it'll, you know, spit

out a certain amount, but then it stops.

The copywriter has tools to do, you know, kind of big things.

Some of them, some of them have extra knowledge and are trained.

We're very gradually ramping up, content ingestion for some of our experts so

that they get like, basically we can subscribe them to newsletters and then

the content of the newsletter gets, digested and then put into their

knowledge base so that they have like kind of special knowledge.

We also have human augmented AI.

Most of our assistants are just pure AI constructed characters, but a subset of

our assistants actually have a human, they're clones of a human that is a

particular expert in, you know, something.

So like for instance, Chandel Stone, who is well known here in Mexico city,

because she's, she runs a startup accelerator is kind of like our

resident startup advisor, so her bot is trained on a bunch of her, you know,