Within the study of machine learning, you'll often hear that the objective is to find the solution that maximizes likelihood. We have a statistical model that is designed to estimate the probability that a certain datum is encountered. We would like for our model to deem that the data we show it is maximally probable, and for other, extraneous data to be less probable. In optimizing for this objective, we can recover a picture of the distribution of our data.
For instance, if we have a bag of marbles that are 50% red and 50% blue. We would like our model to learn that the probability of seeing a red marble is 50% and a blue marble, 50%. Meanwhile, we'd minimize the estimated probability of a green marble occurring down to 0%, given that this will never happen.
Probability estimates are a finite resource that we must partition across all outcomes, and we want to maximize estimates for the the things that do truly happen within our dataset. Putting too much faith on one outcome means underestimating another, which will prompt adjustment. If done successfully, the maximization will naturally share probability mass in such a way that matches the observations.
In our case, the model effectively becomes a "model" of this bag of marbles. We could then perform simulations with our model, taking digital samples of marbles and find that it too yields marbles of colors matching the same probabilities as the bag, symbolically representing the process of reaching into the bag and grabbing one.
While maximum likelihood is a perfectly fine way to describe what's happening, I've often liked to describe our predictive model as following the minimum faith belief. Similarly to how I've described assigning probability mass, faith is also a finite resource. The two are equivalent, but I think it offers a nice perspective and way to talk about it.
The belief is a strategy that optimizes for having to make the fewest and smallest leaps of faith possible. It is a conservative approach, remaining in uncertainty as much as possible, and only committing to an outcome when that confidence can truly be warranted. If "wrongness" could be measured by how far off our estimates are and the net regret we'd feel, we are aiming to be the least wrong. However, it also will never bet and be "extremely right"
On a coin toss, we would remain at maximal uncertainty of 50/50. We would neither extend more faith to the heads or tails outcome due to what we know of the random toss of a coin and the limited evidence we have to work with that would strongly bias one way over another.
For estimating whether it will rain or not in the next hour, we have some correlating, predictive variables that can reduce uncertainty. For instance, historical data might have shown us that a certain combination of air pressure and present moisture conditions can suggest the weather an hour from now with at least some precision. This time around, the maximally uncertain belief of 50/50 would lead us to being wrong more often than we would need to.
It doesn't require faith from us to suspect that a drop in air pressure suggests a higher probability of rain. Not that it will rain, but that it is more likely.
What would require faith is believing strongly that it will stay dry despite conditions suggesting otherwise. What would require faith is the belief that the next marble we pull will be red with 80% probability despite a historical 50/50. What would require faith is overestimating the chances of you winning the lottery.
Faith then, is an abstinence from reason.
Either a deliberate abstinence where we have the true probabilities given to us, or a softer abstinence where our guesses of probabilities are estimates based off of what has happened historically and what all of our experiences, our priors, have informed us, but are biased beyond what is "reasonable".
We can't exactly be faulted for the latter. There is much that is ambiguous or even presently intractable. What is reasonable to you may not be reasonable to me, and vice versa.
This is particularly hard to observe as well. Probabilities are recognized implicitly. But in the space of the world, we depend on our gut, which can be just as much rewarding as damning. We commit to an outcome because that's what life asks of us. If we couldn't commit to anything, we'd do nothing.
We might have a 1-10 feel as to whether a stock price will move up or down in the next week, but ultimately we need to make a decision as to whether we will participate for the ride or not. For the game of life, our estimates amount to nothing useful unless we can use them to take tangible actions. You have to sample from your distribution. Though philosophically we can remain in inaction and ponder on all the reasons it is right to assign a bull market with 80% probability and bear market with 20%.
Ironically, well-fit statistical models are the least biased estimators we can obtain by design. And by definition, neural networks, too, are incredibly low bias, especially in the age where scale is preferred over inductive biases. I say this is ironic because of how common accusations of neural networks showcasing biased predictions, often around racial or political matters.
But these claims hold weight. Taking a set of generated outputs quickly reveals that they do not accurately reflect the world. So where did we go wrong?
In most cases, the issue is that the training data is not representative of the real world, and similar can be said for us humans, seeing only the slice of the world that is before your eyes. So even if we fit to the data in a nicely unbiased manner, the data itself is often biased, and unfortunately, this is all too common.
The distribution of images scattered on the internet are biased by who has access to the internet, who runs publishing agencies and meme accounts, and a self-perpetuating culture that breeds a distortion of reality. Broadly speaking, the internet is a symbolic representation of the real world that has been so far abstracted that it has lost touch with reality.
The models themselves are merely mirrors. Intact, clear reflections that highlight that the sources of our data are skewed.
Our networks have taken no leaps of faith, for they would otherwise risk a greater error over their training data than needed. You kinda have to respect them for that. We have an arsenal of cognitive strategies like pride, wishful thinking, and beyond that all offer utility but at the cost of an unbiased read of our world.
This perspective of minimum faith has shaped my thinking a lot. I recognize that it may be an unreachable ideal as a human and having a limited set of accessible observations of the world seen through a personal egocentric filter. Though I think its something to strive gently for.
Much how like Descartes relinquished all his beliefs to begin again tabula rasa, establishing first "I think, therefore I am." I aim to limit what I can say I know for sure, refrain from prematurely placing unwarranted certainty on hypotheses, make explicit what my beliefs are conditional on, and ultimately find peace in not knowing.
One such example of this lies in the belief of a higher power. I have never had strong feelings here, though through time I have developed into something between deism and agnosticism.
The only guiding factor here is the recognition that cause-and-effect has applied universally within the time I've spent existing and it feels reasonable enough to trust that it has long been a fact of our universe. Because of this, I believe there has to have been a first cause to put everything else in motion, but as to what exactly that first cause is, I have no idea. I have no further assumptions.
Conditioned on the universe being a tangible thing that does obey the laws of physics we know and not all being an illusion, this, to me, feels like the minimum faith belief. It is about as high breadth and vague as it gets while still making some kind of claim, and even then I don't put all my chips on this theory.
To start building a more complex story, like a benevolent god with human-like traits, or equally, to believe there is nothing out there at all despite the fact that something seemingly would have had to set the domino effect of the world in motion, feels like it only requires greater leaps of faith.
All things may as well be equally possible given how much this subject leans into the intangible and supernatural. It really is anyone's guess.
Though by Solomonoff Induction, the greater the complexity a hypothesis takes on, and the more it asks you to subscribe to, the less likely it tends to be, in the scope of a world that has functioned by this rule up until now.
I like to explain this principle with an example of finding cookie crumbs on the kitchen floor. You know that your brother loves cookies and also tends to be a bit messy. It would be reasonable to suspect he is the cause. Though he claims someone broke into the house and caused the mess. Somehow also managing to leave no trace of breaking and entering, making any sound... and going after cookies of all things? This is of course ridiculous. Ridiculous, but possible. Possible, but not plausible. It takes a good bit of faith to subscribe to this explanation, and in how much faith you would put in your brother's word.
There is a good bit of mental play that opens up when you consider how much you do not know and sit with that uncertainty. Possibilities arise that we can still bring to mind and speculate why they came up for us in the first place, even if they are unlikely in actuality. Instead of immediately committing, viewing your extensive gallery of hypotheses that we may normally never even notice and questioning why we have given a certain level of probability to each tells us a lot about ourselves, our desires, and our learned priors. In doing so, I think we can decouple the parts rooted in logical, precise modeling invariant of the world's dynamics invariant to the person from the biases who make us who we are. If we all managed to learn the same true model of the world, we might all be the same person!
Also, I don't just mean dissecting our estimates on predicting weather conditions or stock prices. This extends as well to how we how we imagine others to react, or even how we expect ourselves to act or feel in response to something, illuminating how multifaceted we all are.
I'd remark though this is a mental exercise and perhaps a useful reflection more than anything. Our biases define our unique experiences, and having faith is not a bad thing; it's what makes us human.
Comments