I have previously written about how I work with AI and the kinds of things I do with it. Even since I last wrote about that in February, things have changed. A lot. In February, I described creating a hacked-together remote LLM prompting tool to enable me to trigger ongoing use of my LLM while I was away. I didn’t use it much, just occasionally, because it was hacky and required me to set it up in each project I wanted to use it for (by design – for security reasons). But, then remote use came out for most AI agents, and my hacked-together-solution was no longer needed. My use of AI for “laptop tasks” went up dramatically, while my time spent at my laptop physically went down. This was a huge win for me because for various reasons sitting at my laptop is not always feasible. Now, I could be out walking or hiking and keep projects running. I could also be sitting in a different room and also keep projects running, even with my laptop not in reach. This was a big and delightful change for me.

The development of coding ‘agents’ that can go off and do a task as directed, more reliably than before, has also been a huge change and benefit in the last six months, too. (And now, sub-agents! One agent can direct others to run in parallel.)

If you haven’t given it a try, I highly recommend you try some of the coding agents and use them remotely from your phone to see what they can do and what situations you might benefit from using them in.

But it’s not just “work” that I’m doing more of or differently. I’m branching out to solve more and more of my software-shaped-feeling problems. I continue to build custom apps (like Scheduler Pilot and Protocol Pilot – the latter of which is available if anyone else needs/wants an app to help with tracking and titrating medications) at the drop of the hat. Collectively, I have over half a dozen custom apps I use *daily* which really surprises me (that I have and use so many). Some I only need to use once a week or so, but several I use daily.

I’ve also been able to drastically improve other existing custom apps I have. One big example is Baseline Pilot, which I built last year to enable Scott and me to share our biometric data every morning. For the last ~10 months or so, that has involved us manually opening the app and hitting a button to export the data, and the other person clicking it and opening the app to load it in. It was revolutionary at the time, but it required manual effort on both sides. Recently, though, I had to deploy a server for a work project (which, hello imposter syndrome, I’ve never done before) and do all kinds of technical stuff that I am hugely intimidated by (or – was!) and was still able to do. Because of that, I lowered the friction / barrier to entry for other projects to use a server for data storage and data transfer. As a result, I ended up building a family-focused 3D printing management app so that multiple people in the family could submit files to the person in the family who has a 3D-printer (and so they could access the files in one place and put them in queue etc). That was intimidating, but I had figured out how to do that, so the next thing on my mind after work use and the 3D printing management app use was to then think about if this would solve my manual-click burden for Baseline Pilot. And the answer was yes. Using the same infrastructure that I had set up for the other projects, I was able to quickly (less than an hour) get Baseline Pilot set up to load data into the database in the server and pull down to my and Scott’s apps – automatically – without having to click to share data every day (and click to import the other person’s data every day). It…just works! I open Baseline Pilot and see Scott’s data, without him ever clicking and exporting the data. Woohoo! It thrills me every day for the past several days to open the app and see the data. (And yes, I have it set up where it will send me a push notification for any data above the cutoff I specified, eg >3 standard deviations from normal on key metrics).

I also built a new app called RoutePilot, which allows someone to pull in completed workouts from Apple Health and see if any run intervals were detected. This way, someone learning to run and doing run/walk intervals can track how much running they did without having to hit things on their watch or phone to manually track it while they’re doing it. Because I built this and was showing it to Scott, he realized some of my visuals gave him ideas for how he might use some of this to create cool displays to show pace versus distance PRs of different segments, both for retrospective analysis (my app’s original intent) but also live during a run. This then evolved as we realized this was useful for our serial hiking when we repeat hike routes and find it motivating (sometimes) to know if we are near or above our previous best for a route, so we added in some neat hiking features too. Just yesterday while out hiking, we were doing a “new” route, where we did a hike we’ve done before as a loop but in the reverse direction. Scott wasn’t aware on the way down/back which connecting trails we were taking. It wasn’t as easy as he wanted to be able to adapt the route we had plugged into the app. I suggested being able to load the trail maps and tapping the segments to quickly build a route, that way we could see it’s distance and elevation. He set the agents off and by the time we were home, it was built. Today, I picked up his code and worked on it some more, solving some bugs to allow me to connect segments in different ways, improving the look of it, and adding additional features to this. This type of project used to take us months. Now it’s minutes/hours/days instead of days/weeks/months.

It’s very cool to use AI to help me learn to do new things and then be able to take those skills or awareness of what is possible and apply it to more and more projects. Plus, because I am getting more done on ‘work’ and ‘personal projects’ (using quotes because as an independent researcher there’s a lot of overlap, some unfunded projects are also work), I have more momentum and energy for other things in my life, too.

One surprising example is that I am cooking a lot more. But not just because I have more time. It’s mostly because I have figured out what kind of feedback and input I can get from an LLM to make it easier for me to cook. This actually started by getting Scott to help me cook more. I would figure out what we wanted to make; plan and buy the ingredients (e.g. put them in our grocery delivery); decide what day we wanted to make it; put time or a request for time on the calendar to do it; then tell Scott what I wanted him to prep or get out to help me. Sometimes I’d then do the ‘cooking’ or baking, and other times I’d have him do it all. He gave me feedback on what methods were most helpful – which is a preferred style of listing ingredients in the recipe and certain steps he wanted spelled out. We began using a shared project folder in our LLM of choice where he could see recipes that I customized in the format of his preference. He also liked being able to ask it questions as he went, whether it was converting units on the fly or what cooking instrument to use (e.g. spoon versus whisk, etc). Then at the end, we’d update the recipe for next time to include all the decisions we made and tweaks to the methods or ingredients, so we had a better working artifact (err, recipe) for next time if we made it again. This iterative feedback process made him more willing to do this and it made it easier for me to start exploring other recipes we could make this way, too. I began to be more ambitious with finding varieties of the things we already liked. I even was ambitious and after eating something at a restaurant I really, really liked, had my LLM look up and find the original recipe and then help me adapt it.

(We do a lot of adapting of recipes. First, because I have celiac and I’m gluten free. Second, because we are fairly minimalist and are happy to use frozen pre-chopped or canned vegetables or other ingredients like that when possible. The lovely thing about talking to an LLM to convert a recipe is to say “we have X, Y, Z – use that” even if the original recipe called for something else, or to ask if it B works as a substitute for A or not, and what the tradeoffs are if we used it anyway. This is helping me overcome some of the stigma I faced earlier in life when talking to other people about the hardships I found cooking – some of which was because cooking for 1 person was hard, some of which was because there were far fewer gluten free options or substitutes a decade ago, some of which is because modern technology is awesome and now there’s fantastic options like more canned and frozen pre-chopped vegetables (frozen pre-chopped onions! Frozen cubes of garlic or ginger!) that didn’t exist back when I first started. I have had family members and friends be fairly dismissive when I talked about how I found it challenging to cook the way they did. Which was frustrating because it always felt like a moral failing on my part when in reality I had more barriers and fewer resources accessible to me. Now, the resources are all here – more GF options, more general resources like frozen pre-cut veggies, and LLMs to help me work through recipes and oops I did X instead of Y how do I fix or address that or does it matter? Some of this is AI and some of this is just nice growth in food resources coming together at the same time.)

I am also using AI to create art. Except, put down your torches and pitchforks, please. Not in the way that statement might bring to mind (the way that people have knee-jerk reactions to). I am painting with acrylics on canvas, by hand. But, I use AI quite a bit in the process. I usually find a painting that I like and ask AI to help me adapt a reference painting that is in my skill level (e.g. beginner or advanced beginner), so I have something to work from. Or, I’ll give it a photo from a hike that we did and ask for it to turn it into a reference painting for me. Then, I paint. I take pictures of my canvas as I go and sometimes upload them and ask it for feedback on a specific area or overall, or ask it to give me suggestions on what order to create my painting in. That’s actually how I started, because as someone with no exposure to painting before, I had no idea where to start! I would ask it to give me a step by step high level approach to the painting, and would follow that. (Obvious things like working back to front, starting with the sky etc, in a landscape are not obvious when you are a 100% beginner). Eventually I learned additional ways to get help with my painting. I sometimes ask it to give me advice on fixing a color, or how to mix a color, given the paint options that I have. Other times I’ll ignore the original reference and just tell it to focus on my work in progress picture and give me high level feedback. It’s been incredibly helpful to me and has facilitated me painting >30 paintings in the last year or so since I started painting. It’s really useful for customizing feedback to where I am, skill-wise. I also ask it sometimes to assess my portfolio and give suggestions for what skills to work on or document what progress I’ve made on existing skills I’ve been working on. That’s also helpful for seeing where you do have actual skills already (or strong skills compared to others), so you don’t just feel overwhelmed and inept all of the time. You can drill in or scale out, feedback wise, and you can ask it over and over again if you are stuck on painting something and can’t get it quite right. It doesn’t get bored, it doesn’t (openly) judge you – unless you ask it to, and it’s nice to be able to have something to give feedback and input on the specific things I want input for. It’s better than the next available human, because I don’t have a next available human who is interested in my hobby project. (Scott, who occasionally gives surprisingly prescient feedback especially about color values in a way that my eyes/brain don’t intuit the same way, isn’t always available or interested to the degree that I am interested in this – so LLMs often win for iterative feedback in the absence of an available, interested human).

Both cooking and art are examples of things where AI facilitates me to do a lot more than ever before. It’s less surprising that AI has helped me build all kinds of software projects from apps to drug titration tools and simulators. But being able to use AI to interact more with the real world is a use case that I don’t know that I would have predicted, and it’s exciting to see how else it might enable me in the future. There’s probably half a dozen other physical world interactions it’s facilitating for me, too, that I don’t think of as obvious examples. This includes things like giving me feedback on my strength training work to tweaking and improving my spreadsheet where I log and automate filling in my strength training workouts into my other all-activity tracking spreadsheet…it’s a nice mix of physical world and software interactions that all are making my life easier and better than before.

If you have cool, surprising uses of AI helping you in the physical world, I’d love to hear about them! Send me (or comment) your ideas and experiences.