A Poisson pondering

It might seem counterintuitive, like, how could a sample size be infinite?

But let us remember the classic application of the Poisson distribution — modeling the probability of a given number of events occurring in a fixed time interval.

To better understand it with an example, consider the probability of a given number of buses arriving at a station in an hour.

Now, to look at it from a binomial distribution perspective, we could divide this one hour into sixty Bernoulli trials. Each trial models the probability that a single bus will arrive in that particular minute.

The problem with this approach is that it simply restricts the number of buses that could arrive at any given minute to one. In the real world, it is not a completely unimaginable scenario that multiple buses could arrive back-to-back within a minute.

We could increase the granularity by opting for 3600 Bernoulli trials instead of 60. It would mean we moved away from looking at minute intervals to seconds, and our problem might have an acceptable solution now.

But consider the cases where more than one event could happen within a second, and it becomes clear that we would have to go even more granular, which causes the number of trails (n) to increase again.

We keep increasing this granularity to reach a subinterval where events could occur only once. Then, the n value would increase to such an extent that it approaches infinity.

Hence, we could apply a limit that the n tends to infinity on the probability distribution function (PDF) of Binomial distribution, and we would have the PDF of Poisson distribution. It proves that the Poisson is an extended case of Binomial when the number of trials is large (or large sample size).

Another key distinction of the Poisson distribution is that it assumes the events to be independent with a constant rate of occurrence.

Let us equate this constant rate (famously denoted by a ‘λ’) to the expected value of the random variable if it were a Binomial distribution.

The expected value of a random variable in Binomial distribution = number of trials (n) * probability of success in each trial (p).

Therefore, λ = n * p

Since λ is constant and n approaches infinity, the p becomes very low and approaches zero.

Hence, we could use the Poisson distribution to model the rare events with a low success rate, like people reading my articles and loving them!

Comments

Solving Customer Churn with a hammer!

Learning when data should take a back seat and give way to domain knowledge is a valuable skill. Suppose you built a machine learning model on the data of your customers to predict churn risk. Now that you have a risk score for each customer, what do you do next? Do you filter the top n% based on the risk and send them a coupon with a discount in the hopes that it will prevent churn? But what if price is not the factor driving churn in many of these customers? Customers might have been treated poorly by customer service, which drove them away from your company's product. Or there might have been an indirect competitor's product or service that removes the need for your company's product altogether (this happened to companies like Blockbuster and Kodak in the past!) There could be a myriad of factors, but you get the point! Dashboards and models cannot guide any company's strategic actions directly. If companies try to use them without additional context, more often tha...

Curing writer's block with sunk cost fallacy

I paid $20 to renew this blog's domain in July. But the truth is, I had been suffering from writer's block ever since the start of this year and hadn’t posted a single thing. At one point, I was ready to give up on the blog altogether, but a voice in my head kept reminding me of all the time and money I’d already invested in this blog. So, this week, I sat down to write this imperfect, patchy article—about none other than that voice itself. Let me start with a classic scenario where you might have also encountered this voice. Suppose you’re at an Italian restaurant and ordered some pasta and tiramisu. After finishing the pasta, you realize you’re full, and there’s no way your stomach can handle that delicious tiramisu sitting right in front of you. But then, that beautiful brain of yours reminds you that you’ll be paying for the tiramisu whether you eat it or not. In a desperate attempt to avoid wasting money, you reluctantly eat two quick bites. And just like that, my frien...

What is SUTVA for A/B testing?

Imagine if person B’s blood pressure reading depends on whether person A receives the blood pressure medicine in a randomized controlled trial. This will be violating Stable Unit Treatment Value Assumption (SUTVA) SUTVA states that the treatment received by an individual should not influence the outcome we see for another individual during the experiment. I know the initial example sounded absurd, so let me try again. Consider LinkedIn A/B testing a new ‘dislike’ reaction for its users, and the gods of fate chose you to be part of the initial treatment group that received this update. Excited after seeing this new update, you use this dislike reaction on my post and send a screenshot to a few of your connections to do the same, who are coincidentally in the control group that did not receive the update. Your connections log in and engage with my posts to use this dislike reaction, but later get disappointed as this new update is not yet available to them. The offices of LinkedIn are tr...

The Dumb Datum

Search This Blog