Walt Disney Tried to Warn Us – ΔΕΙΤΕ ΤΟ ΠΡΙΝ ΤΟ ΚΑΤΕΒΑΣΟΥΝ

One of the funniest kinds of YouTube clickbait is “X movie TRIED TO WARN US!” (this is my favorite in English speaking content, in Greek my absolute favorite is “ΔΕΙΤΕ ΤΟ ΠΡΙΝ ΤΟ ΚΑΤΕΒΑΣΟΥΝ” which can be translated as “See it before it is taken down” – there are videos bearing that in the title that are well over nine years old). So, allow me to steal a page from their book.

In Disney’s Fantasia (the seminal 1940’s cartoon) there is a segment where Mickey Mouse is an enterprising Sorcerer’s apprentice. Tasked to carry water, Mickey animates a broom to help him carry the water to the sorcerer’s cauldron. At first, the scheme worked just fine. However, down the line Mickey realises he cannot stop the broom from fetching water, chops it into a gazillion pieces only to find out that they have regenerated and keep bringing in water. “Nice story, Athanasios and thanks for pointing us to one of the greatest cartoon moments of all time but what does it have to do with security?” I can hear some of you thinking (I can  also hear some of you thinking “Great! Yet Another AI related article”).

Allow me to synthesize the following pieces of information. Dr Gary McGraw (of BSIMM fame, one of the few folks thinking of software security when most were thinking of packet filters) and a few other brilliant folks have started Berryville Institute of Machine Learning, a non-profit science and technology research organization focusing on ML security (you can call it AI security). On their “No Security Meter for AI” technical publication in a short 20 pages not only they establish a nomenclature about Machine Learning Security (including calling the field nascent, I really loved that analogy), they contrast the field of “traditional” software security and its evolution in risk management to AI/ML security. Arguably, the discipline has no success meter. I have blogged in the past when you enter MLSec, you have all the problems of Product Security with a whole new amount of problems (be it technical ones or organizational/market ones) but the paper is right on the money – as security practitioners, we need to do better in measurement and systematic risk management.

Another important piece of news is that MITRE and OpenAI joined forces to create a framework (which to their credit, offered up for discussion in the broader MLSec community) for “Classifying AI Control Incidents and Their Severity”. A nice move by all means and it actually augments the argument that BIML makes, let’s start identifying what can go wrong and classify it accordingly.

So, what does all this have to do with Mickey Mouse? Right this moment, we see large scale adoption of technology that is not every time widely understood (once I was asked from a potential client how to ensure their LLM will give the same answer every time, I politely declined to engage), is widely deployed (often in an agentic basis) and might end up either executing what is not supposed to so (or not do what is supposed to) or creating code that powers real life systems. Attempting to measure both orders (first order: direct execution of actions, second order: execution of actions based on ML/AI generated code) is a great first step. Let’s just hope that Akerlof (and general market inefficiencies, deliberate or not) will not stifle said initiatives, lest we end chopping up broomsticks. Obviously, there is a lot of material to cover than a short blog article can cover so expect more material on this coming.

Leave a comment