Probability

What Is Probability?

Probability quantifies the chance that an event will occur. It represents uncertainty and is used when exact prediction is impossible. Probabilities are numbers between 0 and 1

Discrete Outcomes
  • Occur when the number of possible outcomes is finite or countable.
  • Example: Number of heads in coin tosses.
  • Example: Number of people on telephone calls in a given period.
Continuous Outcomes
  • Outcomes can take infinitely many values within a range.
  • Example: Length of time a phone call lasts (between 0 and 10 minutes).
  • Cannot assign nonzero probability to a single exact value in continuous cases.
Probabilistic Modeling

Building a probability model means mapping a real-world situation to a mathematical structure (e.g., random variables, distributions). Models often make assumptions — e.g., independence (one event doesn’t affect another).

Illustration: Probability of k heads in N coin tosses:
P[k] = \binom{N}{k}p^k(1-p)^{N-k}

Analysis vs. Computer Simulation

Traditional analysis uses mathematical formulas to derive results. Computer simulation uses repeated computational experiments to approximate probabilities and build intuition.

Role of Simulation

  • Helpful when analytical solutions are difficult or intractable.
  • Visual or numerical results from simulation help validate or illustrate analytical theory.
Key Takeaways
ConceptBrief Summary
ProbabilityA measure of how likely an event is; values between 0 and 1
Discrete vs. ContinuousDiscrete: countable outcomes; Continuous: uncountably infinite outcomes.
ModelingAbstracting real phenomena into probabilistic terms.
SimulationComplementary to analysis; enhances intuition.
Motivation and Role of Probability
  • Probability provides a mathematical framework to model uncertainty.
  • Used when outcomes are not deterministic but governed by chance.
  • Engineering applications include:
    • Communication noise
    • Signal detection
    • Reliability analysis
    • Decision making under uncertainty
Random Experiment

An experiment whose outcome cannot be predicted with certainty.

Sample Space (Ω)

The set of all possible outcomes. Examples:

  • Coin toss: Ω = {H, T}
  • Dice roll: Ω = {1,2,3,4,5,6}
Events

An event is a subset of the sample space. Types of Events are:

  • Simple event: single outcome
  • Compound event: multiple outcomes
Axioms of Probability

Probability is defined through three axioms. These axioms form the foundation of all probability results.

  1. Non-negativity
    P(A) \ge 0
  2. Normalization
    P(\Omega) = 1
  3. Additivity: For mutually exclusive events:
    P(A \cup B) = P(A) + P(B)
Basic Probability Rules
Complement Rule

P(A^c) = 1 - P(A)

Union of Two Events

P(A \cup B) = P(A) + P(B) - P(A \cap B)

Monotonicity

If (A \subset B), then:
P(A) \le P(B)

Conditional Probability

The probability of event A given B has occurred:
P(A|B) = \frac{P(A \cap B)}{P(B)}, \quad P(B) > 0

Interpretation
  • Updates probability based on new information
  • Core idea behind learning and inference
Independence

Independence means knowledge of one event gives no information about the other. Two events A and B are independent if:
P(A \cap B) = P(A)P(B)

Equivalent condition:
P(A|B) = P(A)

Law of Total Probability

If events B_1, B_2, \dots, B_n form a partition of Ω:
P(A) = \sum_{i=1}^n P(A|B_i)P(B_i)

Used when:

  • A problem can be broken into cases
  • Often applied in detection and classification problems
Bayes’ Theorem

P(B_j|A) = \frac{P(A|B_j)P(B_j)}{\sum_{i=1}^n P(A|B_i)P(B_i)}

Interpretation

Converts prior probabilities into posterior probabilities. Central to:

  • Statistical inference
  • Machine learning
  • Signal detection
  • Medical testing
Subjective vs Frequentist Interpretation
Frequentist View
  • Probability = long-run relative frequency
  • Requires repeated trials
Subjective (Bayesian) View
  • Probability = degree of belief
  • Can be updated using Bayes’ rule
  • Strongly emphasized by Kay for engineering intuition
Key Takeaways

Probability is axiomatic but intuitive. Conditional probability is the most important concept. Bayes’ theorem is fundamental to modern engineering systems. Understanding probability is essential before studying:

  • Random variables
  • Random processes
  • Noise models

Random Variables

References

  1. Introduction of Intuitive Probability and Random Processes Using MATLAB by Steven M. Kay (Springer, 2006)