The dark secret of AI is that the models are black boxes. We can see what goes in (to an extent) and what comes out. In between, all accessibility to human insight, interpretation, and accountability is blacked out. And it starts with tokenization. Tokenization is the process of taking letters, words, paragraphs (or numbers, equations, and proofs) and chunking them into clumps based solely on commonness. These tokens literally break up interpretable, meaningful representations of things, thoughts, and values into meaningless chunks that occur more frequently than other meaningless chunks. There is nothing intelligent about “AI” tokenization. The modelling then takes strings of these meaningless chunks and predicts other strings of these meaningless chunks based solely on the most common correlations between strings of meaningless chunks of data. The only thing intelligent about the process is at all is that humans have generated so much electronic data, on so many topics, in such depth that useful outputs can often occur despite AI modeler’s best attempts to break up and mask any meaning in the data or models themselves.

…

LIGHT BOX RESEARCH (LBR) will be the opposite. Data segmentation will highlight features and integrate context, modelling approaches will illuminate underlying mechanisms, and model out put will be reliable and reproducible. At every step of the process, LBR will strive to enhance human understanding of and interaction with the data, not try to replace it.

…

You are here > Intelligent modelling begins with intelligent segmentation.