Basic Concepts

Basic Concepts

Tags
Statistics
Date
Sep 25, 2019
Description
Introducing the concept of the probability of an event. Also covers set operations and the sample-point method.
Slug
mathematical-statistics-basic-concepts
In this chapter, we introduce the concept of the probability of an event. Then we show how probabilities can be computed in certain situations. As a preliminary, however, we need to discuss the concept of the sample space and the events of an experiment.

Experiment, sample space and events

An experiment is the process by which an observation is made. In particular, we are interested in a random experiment whose outcome is not predictable with certainty. The set of all possible outcomes of an experiment is known as the sample space of the experiment, and is often denoted S. An event (denoted ) is a set that contains some possible outcomes of the random experiment.
By definition, any event is a subset of the sample space. For a given random experiment, its sample space is unique. Let’s see some examples.
Experiment
Possible outcome
Sample space
Event
Test of a certain disease on a patient
Positive or negative
Rolling a six-sided die
Outcome
Tossing two coins
Each coin is either head or tail
First toss is a head:
Life time of a computer
All non-negative real numbers
Survives for more than 10 hours:
For each experiment, we may define more than one event. Take the die-rolling example, we can define events like
If we observed the event , it means we observed one of the three events , or . We say can be decomposed into , and . If an event can be further decomposed, it is called a compound event. Otherwise, it’s called a simple event. Each simple event contains one and only one outcome.
Finally, events with no outcome is called the null event, and is denoted . For example, an event of the outcome is greater than 7 in the die-rolling example.

Set operations

Suppose we have a sample space and two events and .

Subset

If all of the outcomes in are also in , then we say that is contained in , or is a subset of . We write it as . Subsets have several properties:
  1. Any event is a subset of the sample space: .
  1. Any event is a subset of itself: .
  1. If and , then .
  1. , .
  1. .

Union

We denote as the union of the two events. It is a new event which consists of all outcomes in , and all the outcomes in . In other words, . The union operation also has a few properties:
  1. .
  1. .
  1. .

Intersect

We denote , or for short, the intersection between and . consists of all outcomes that are both in and in . Its properties are straightforward:
  1. .
  1. .
  1. .
We say E and F are disjoint or mutually exclusive if . Any event is disjoint with the null event.

Complement

Finally, for any event , we define a new event , referred to as the complement of . consists of all outcomes in the sample space that are not in . Its properties include
  1. and .
  1. .
  1. The event is always disjoint with its complement: .
  1. .

Example of set operations

❓
Consider rolling two six-sided dice. Let
and we want to find (1) , (2) , and (3) .
The sample space is
and the events are
We skip for now as it’s more complicated. We can find that and only have one element in common, , so , and .
Finally, we can express the complement of as , which is , and we can write down all the possibilities as .
A graphical representation that is useful for illustrating logical relations among events is the Venn diagram.

Laws of set operations

The operations of sets can be applied to more than two events, and they follow certain rules similar to the rules of algebra. All the following rules can be verified by Venn diagrams.
  • Commutative laws: , .
  • Associative laws: , .
  • Distributive laws: , .
In addition, there is a law that connects all three operations (union, intersection and complement) together, DeMorgan's law:
DeMorgan’s law can be extended to more than two events. Let denote the union of events to , and their intersection,

Probability of events

One way of defining the probability of an event is in terms of its relative frequency. Suppose we have a random experiment with sample space , and we want to assign some number to represent the probability of event . We may repeat this random experiment many times. Let be the number of times in the first repetitions of the experiment that the event occurs. The probability of the event is defined as
There are a few drawbacks to this method:
  1. It requires to be countable.
  1. We need to assume that the limit exists, and is a positive number.
  1. Sometimes our random experiments are limited and we can’t repeat it many times, or the experiment may not even be observable.
To overcome such drawbacks, modern mathematics used an axiom system to define the probability of an event.

Axioms of probability

For sample space and event , we define three axioms.
Axiom 1
Axiom 2
Axiom 3
.
.
For any sequence of mutually exclusive events
where for any and where (mutually exclusive). More formally, we can say to be -additive.
The definition through axioms is mathematically rigorous, flexible, and can be developed into an axiomatic system. We’ll show the flexibility through an example. Suppose our experiment is tossing a coin. If we believe it is a fair coin, we have
then using Axioms 2 and 3 above, we can derive
so . If we believe the coin is biased and , then
By combining the two equations, , .
In this example, we didn’t use any information of the observations or frequencies. We are assigning probabilities according to our belief so long as this assignment satisfies the three axioms. Based on the axioms, we can prove some simple propositions of probability.

Propositions

Proposition 1
. The proof is given as follows.
Proposition 2
If , then P(E) ≀ P(F). To prove this, note that and are mutually exclusive.
Venn diagram of Proposition 2.
Venn diagram of Proposition 2.
Proposition 3
. This proposition can be easily proved using a Venn diagram. Let , and denote , and , respectively.
Venn diagram of Proposition 3.
Venn diagram of Proposition 3.
❓
Now let’s apply these propositions to the example below. A student is applying for two jobs. Suppose she’ll get an offer from company A with probability 0.3, and an offer from company B with probability 0.4, and with probability 0.3 she gets both offers. What is the probability that she gets neither offer?
We have and . Knowing that ,  and

The sample-point method

As somewhat shown in the example above, for an experiment with finite or countable number of outcomes, we can calculate the probability of an event through the so called sample-point method. The procedure is
  1. Define the experiment, sample space and simple events (outcomes).
  1. Assign reasonable probabilities to each simple event.
  1. Define the event of interest as a collection of simple events.
  1. Calculate the probability of the event by summing the probabilities of the simple events in the event.
The main idea of the sample-point method is based on Axiom 3.

Coin flip example

❓
A fair coin is tossed three times. Find the probability that exactly two of the three tosses are heads.
We’ll follow the procedure in the sample-point method. The experiment is β€œtossing the coin three times”. The sample space is
where each of the 8 outcomes can be considered as a simple event . Since we consider it as a fair coin,
Our event of interest, , is defined as {2 heads and 1 tail}, so
The final step is calculating the probability of
Note that in this case (and in many other experiments), all the outcomes in the sample space are equally likely to occur. For such experiments, we can simplify the sample-point method as

Powerball example

❓
The Powerball is one of the largest lottery games in the US. The system works like this:
  1. 5 numbered white balls are drawn out of 69 balls without replacement.
  1. 1 numbered red ball is drawn out of 26 balls.
You win the Powerball if you chose exactly those 5β€…+β€…1 balls, and the order of the white balls doesn’t matter. What is the probability to win a Powerball?
It’s reasonable to assume each outcome will be equally likely to occur. Each outcome is a set of 6 numbers satisfying the above rules.
If we draw the white balls one by one, we have ways (ordered outcomes). For each set of 5 numbers, we have ways of arranging them. So the number of ways to draw 5 white balls of 69 is
Formally, this is β€œchoose from ”, which can be written as and
Now we have