%Initial setup
\documentclass[11pt]{article}
\title{EPSE592 Assignment \#1}
\author{Ed Kroc}

%Math Packages
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amsthm,mathabx}

%General formatting Packages
\usepackage{fancyhdr} %To put headers in
\usepackage{multirow} %for tables
\usepackage{graphicx,graphpap,rotate,geometry,subfigure} 
\usepackage{enumitem} %Permits more customisation of lists than enumerate
\usepackage{tikz} %for textcolor

%Document Layout
%Text positioning
\marginparwidth 0pt %
\marginparsep 0pt  %distance between marginal notes box and main text
\oddsidemargin  0pt
\evensidemargin  0pt
\topmargin   0pt
\textwidth   6.8in %control width of text on 8.5X11 page
\textheight  9.50in %control height of text on 8.5X11 page
\voffset -0.8in

\newcommand{\pr}{\text{Pr}}
\newcommand{\e}{\mathbb{E}}
\newcommand{\var}{\text{Var}}
\newcommand{\sd}{\text{SD}}

%Header of the pages
\pagestyle{fancy}
\lhead{EPSE 592}
\rhead{Assignment 1}
\setlength{\headheight}{14pt} %to make room for the header vertically due do squashed height
\fancyhfoffset[R]{0.2in} %to stretch header to match the length of the textwidth.


%=============================
\begin{document}

\begin{center}
\textbf{WINTER 2019/20 TERM 2  \,\, EPSE 592: ASSIGNMENT 1 \\
Due: Week 4 in class: Jan. 30th}
\end{center}

\vspace*{2mm}
\begin{itemize}
  \item
  Please make sure you write your answers to these questions in your own words. Even if you work with a group to formulate your responses, do not just copy someone else's sentences/words. 
  \item
  There is no need to record more than 3 decimal places for any of these problems.
\end{itemize}


\begin{enumerate}[label=\textbf{Question \arabic*:},start=1]

%Question 1:
%===================================================
%%Research design questions: randomization, controls, etc.
\item {\bf (Randomization, control, and confounders)} Suppose you are part of a team that has designed a study to assess the potential effects of social media exposure on anxiety levels in people aged 18-34. Your team will submit repeated calls for participants across the UBC campus until you have enlisted 9 people who report high levels of social media use, 9 with medium levels of use, and 9 with low levels of use. (Suppose the definitions of ``high/medium/low levels of use" are well-defined and uncontroversial.) Each subject will be compensated financially for agreeing to participate in the study.

Once these 27 participants are secured, they will each complete a 10-minute questionnaire to assess their baseline level of anxiety. Afterwards, they will receive a random assignment to one of three treatment groups: low, medium, or high social media use. Each subject will then have to adjust their social media use accordingly for the next 24 hours, afterwhich time they will return to your lab and complete another 10-minute questionnaire assessing their post-treatment anxiety levels.

\begin{enumerate}
	\item Does this study design use random sampling? Is the treatment of interest (social media use) randomly assigned to experimental units? Could you foresee any potential problems with implementing the proposed randomizations?

	\item The study design recognizes that baseline social media usage could be a significant confounding variable; i.e. if higher social media usage is correlated with higher anxiety levels, then individuals who regularly spend lots of time on social media will likely already have higher levels of anxiety when entering the study. Can you think of any other possible confounding variables that might affect our results given how our team has decided to sample experimental units or assign experimental treatments?

	\item Does this study have a control group? If so, who is in the control group? If not, why not?

\end{enumerate}



\vspace*{3mm}



%Question 2:
%===================================================
%%Conditional probabilities and p-values
\item {\bf (P-values, priors, posterior believability, and the truth)}
In this sequence of problems, you will get some practice thinking about the different kinds of conditional probabilities that scientists must consider in their analyses. You will also explore the relationship between p-values (the most common quantitative tool for inference) and evidence for the likelihood of a hypothesis of interest (what we actually care about when we do science). While this discussion could get very complicated very quickly, we will explore the issues with a simple, though entirely representative, example. 

Suppose I claim to have extrasensory perception (ESP); i.e.~the supernatural ability to see things without actually witnessing them for myself. I challenge you to test my claim and so you devise a simple experiment: you will randomly draw one playing card from a well-shuffled standard deck of 52 cards, making sure I cannot possibly see the card, and then ask me what card you drew. This experiment has two possible outcomes: (1) I guess your card correctly, or (2) I guess incorrectly. The null hypothesis (i.e.~the hypothesis of no effect) is that I do {\em not} have ESP, and thus, my guesses are random.
\begin{enumerate}
	\item Let $Y$ denote the event that I guess your card correctly, and let $N$ denote the event that I guess incorrectly. Assuming the null hypothesis, what is the probability that I guess your card correctly: $\pr(Y\ |\ H_0)$? Given the null, what is the probability that I guess your card incorrectly: $\pr(N\ |\ H_0)$? 

	\item Suppose we run the experiment once and I guess your card correctly. The p-value for this observed outcome is what you calculated above as $\pr(Y\ |\ H_0)$; i.e.~the probability of observing an experimental outcome as or more extreme than the one we actually did observe, given the null hypothesis. Since this experiment has only two possible outcomes, and we observed the ``more extreme" one, the p-value is simply $\pr(Y\ |\ H_0)$. Using the typical rule of thumb that {\em a p-value of less than 0.05 is ``significant" (i.e.~decent evidence of an effect)}, is the p-value for this particular experimental outcome significant? If so, would this experimental outcome convince you that I probably have ESP?

	\item Recall that $\pr(H_0\ |\ Y) \neq \pr(Y\ |\ H_0)$. The latter probability is our p-value, while the former probability is what we actually need to evaluate if we want to make an informed {\em decision} about the believability of $H_0$ given the observed evidence; this former probability is called a {\em posterior probability}. [Go back and review the Sally Clark case: $\pr(Innocence\ |\ Evidence)$ is exactly a posterior probability.] There is a famous mathematical result (Bayes' Theorem) that allows us to relate our p-value to the posterior probability of interest:
\begin{equation*}
	\pr(H_0\ |\ Y) = \frac {\pr(Y\ |\ H_0)\pr(H_0)}{\pr(Y\ |\ H_0)\pr(H_0) + \pr(Y\ |\ H_A)\pr(H_A)},
\end{equation*}
where $H_A$ denotes the {\em alternative hypothesis} to the null. For our example, $H_A$ would be the hypothesis that I {\em do} have ESP. The quantities $\pr(H_0)$ and $\pr(H_A)$ are called {\em priors}; they represent how likely each hypothesis is {\em before} we conduct our experiment to test them.

Given everything that we know about general physics, chemistry, and biology - and given all the failed attempts at proving the existence of ESP in other people in the past - it is quite reasonable to assign a very large value to $\pr(H_0)$. Let's say, conservatively, that $\pr(H_0) = 0.999$. What then must be the value of $\pr(H_A)$? Moreover, what is $\pr(Y\ |\ H_A)$; i.e.~what is the probability that I identify your card correctly given that I actually do have ESP?

\item Now we can put all the pieces together to calculate the posterior probability we are really interested in. Use Bayes' Theorem (equation above) and your results from parts (a) and (c) to calculate $\pr(H_0\ |\ Y)$. Is the fact that I guessed your card correctly during the experiment convincing evidence that I have ESP?

\end{enumerate}



\vspace*{3mm}

%Question 3:
%===================================================
\item {\bf (The effect of experimental replication and the importance of critically assessing your quantitative inferences)}
Suppose that I challenge you to test my ESP claim again. We repeat the same card experiment as before, and this time I again guess the correct card. Note that $H_0$, $H_A$, and the p-value $\pr(Y_2\ |\ H_0)$ are all the same as in the first experimental run. Here, we use the subscript to denote which experiment we are talking about: the first or the second. The probability of the event $Y_i$ under the null is unchanged by the value of $i$, however.

The first experiment resulted in me correctly identifying your card: the event $Y_1$. We would like to combine that knowledge with the information we have gained from the second experiment to make a more informed assessment about the believability of $H_0$. 
\begin{enumerate}
	\item Before we ran our first experiment, we reasoned that the {\em prior} believability of our null hypothesis was about $\pr(H_0) = 0.999$. In part (d) of the previous problem, you ``updated" this probability given the outcome of the first experiment: $\pr(H_0\ |\ Y_1)$. So if we now want to update our belief in $H_0$ given the outcome of the {\em second} experiment, we can take $\pr(H_0\ |\ Y_1)$ as our new {\em prior}, since it encodes all the information about the first experiment {\em and} our prior assessment of the hypothesis before we ran any experiments. Applying Bayes' Theorem again, we can compute the posterior probability as
\begin{equation*}
	\pr(H_0\ |\ Y_2) = \frac {\pr(Y_2\ |\ H_0)\pr(H_0\ |\ Y_1)}{\pr(Y_2\ |\ H_0)\pr(H_0\ |\ Y_1) + \pr(Y_2\ |\ H_A)\pr(H_A\ |\ Y_1)}.
\end{equation*}
Write down the prior believability of the null hypothesis, $\pr(H_0\ |\ Y_1)$, and use this value to find $\pr(H_A\ |\ Y_1)$. Write down the p-value of the outcome of the second experiment, $\pr(Y_2\ |\ H_0)$. Finally, write down $\pr(Y_2\ |\ H_A)$.

	\item Putting the pieces together again, calculate $\pr(H_0\ |\ Y_2)$. How does this value compare to $\pr(H_0\ |\ Y_1)$ that you found in part (d) of the previous question? Is the fact that I guessed your card correctly in {\em both} experiments convincing evidence that I have ESP?

	\item You may still be skeptical, so suppose I challenge you one last time to test my ESP claim. We repeat the same experiment a third time, and yet again I guess the correct card. Repeat the updating procedure we performed above to calculate the new posterior probability $\pr(H_0\ |\ Y_3)$.

	\item How convinced are you that I have ESP given the outcome of the three experiments and the quantitative evidence in part (c)? {\em Think critically, not like you are following a recipe} (quantitative inference is {\em not} a recipe to be followed). If you are still skeptical about my claim of ESP, are there any other more reasonable explanations for my ability to guess your card correctly three times in a row?
\end{enumerate}


\vspace*{3mm}

%Question 5:
%===================================================
\item {\bf (T-tests: heteroskedasticity means loss of power)}
In this exercise, you will perform some t-tests and investigate how inequality of variances across test groups can affect your results.

Suppose you are trying to compare average test scores between two sections of the same class. The data are presented in the following table (tests were scored out of 50 points).

\begin{table}[h!]
\begin{center}
\begin{tabular}{| c | c | c | c | c | c | c | c | c | c |}
\hline
Section I 	& 22 & 23 & 19 & 24 & 20 & 20 & 26 & 19 & 21 \\
\hline
Section II 	& 12 & 22 & 29 & 42 & 47 & 33 & 24 & 19 & 38 \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{enumerate}

\item Calculate the sample mean and sample standard deviation for the Section I scores. Do the same for the Section II scores. Informally, does there appear to be a difference in the sample means? Does there appear to be a difference in the sample standard deviations?

\item Apply a formal independent samples t-test to test the hypothesis that the mean scores of the two sections are equal. Report your p-value and interpret. 

\item Apply a formal test to check the equality of variances assumption for your above t-test. In Jamovi, you can produce a simple $F$-test (called Levene's test here) by clicking in the appropriate box in the ``Assumptions Check" options. Report the p-value of this test and interpret.

\item Now suppose you have data on a third section of the same class, given by the data below.

\begin{table}[h!]
\begin{center}
\begin{tabular}{| c | c | c | c | c | c | c | c | c | c |}
\hline
Section III 	& 25 & 26 & 29 & 34 & 32 & 33 & 27 & 29 & 31 \\
\hline
\end{tabular}
\end{center}
\end{table}

Calculate the sample mean and sample standard deviation for the test scores from this section. How do these compare to the same statistics for Sections I and II?

\item Apply an independent samples t-test to test if the mean scores from sections I and III are equal. Report the p-value and interpret. Again, check the equality of variances assumption formally. 

\item What kind of effect(s) did hetero/homoskedasticity have on your analysis and interpretations?


\end{enumerate}

\vspace*{3mm}

\end{enumerate}
\end{document}





%\vspace*{3mm}

%Question 4:
%===================================================
%\item {\bf (Your experiment may not test the hypothesis you think it does)}
%Think again about the results of the previous two problems. We designed and ran (repeatedly) an experiment to test my claim of ESP. The evidence that was observed - three times I successfully identified your card - seemed to suggest, ostensibly, that maybe I do possess supernatural powers. 

%However, other explanations are possible. In particular, it is possible that I could have {\em cheated} at the card-guessing experiment, perhaps by using mirrors or hidden cameras. We can use the evidence collected in the three iterations of our experiment to test the null hypothesis that I was {\em not} cheating versus the alternative hypothesis that I was cheating: denote these hypotheses by $C_0$ and $C_A$, respectively.

%\begin{enumerate}

%\item Let $T$ denote the test statistic that counts the number of times I correctly identify your card in three hypothetical (and independent) iterations of the experiment. What is the distribution of $T$ under $C_0$? That is, what kind of random variable is $T$ when we assume $C_0$ holds? [Hint: it is one of the random variables with a special name we talked about in class.] What are the values of the parameter(s) that define the distribution of $T$ given $C_0$?

%\item Since I actually identified the correct card in all three iterations of the experiment, we know that $t_{obs}=3$. Calculate the p-value associated with this observed outcome. That is, find $\pr(T \text{ is as or more extreme than } t_{obs}\ |\ C_0)$. Note: in this case, $t_{obs}$ is as extreme as $T$ can ever be, so the expression for the p-value reduces to $\pr(t_{obs}\ |\ C_0)$. Interpret the meaning of this p-value in the context of the hypotheses we are testing.

%\item The p-value you just calculated should suggest that the observed data are more consistent with either $C_0$ or $C_A$. But what about the actual likelihood that I cheated, given the evidence? In order to use Bayes' Theorem, as before, to calculate this posterior probability, we need to assign values to the prior believability of the hypotheses $C_0$ and $C_A$. Here, $C_0$ is the hypothesis that I did not cheat. Let's say you think I'm a trustworthy person, so you want to give me the benefit of the doubt: assign $\pr(C_0) = 0.9$. Find $\pr(C_A)$ and $\pr(t_{obs}\ |\ C_A)$; assume that if I cheated, then I cheated every time. Use this information and the p-value you calculated in part (b) to compute the posterior probability $\pr(C_0\ |\ t_{obs})$ using Bayes' Theorem.

%\item Try to visualize the relationships between the evidence and the four hypotheses $H_0$, $H_A$, $C_0$, and $C_A$ that we have been working with by drawing a Venn diagram. Use the notation $E$ to denote the observed evidence; i.e. three independent experiments where I correctly identified your card. Remember that you have values for $\pr(H_0)$, etc. This will help you decide how big to make each piece of the Venn diagram.

%\item Having me guess a card that you randomly draw seems to be an experiment that is better designed to test $C_0$ versus $C_A$ rather than $H_0$ versus $H_A$. Can you think of some ways to improve the experimental design so that we could be more confident that what we are actually testing is $H_0$ versus $H_A$?



%\end{enumerate}



%===================================================


