Let’s start with $Y = X$, then the regression line is a perfect fit. The points of such a dataset is $(1,1),(2,2),(3,3),(4,4),(5,5)$

Adding some normal white noise to these points $(1,1),(2,3),(3,5),(4,5),(5,5)$. A regression line fit on these points will move up. Hence the coefficients of $Y = mX+c$, $m$ will increase, $c$ might still stay at 0 or at max increase.

This movement will go in the negative direction if we predict $X$ based on $Y$

Linear Regression as per me can be used in Time Series but might not always give good results. Few reasons which come up are:

• Linear Regression is good for intrapolation but not for extrapolation so the results can vary wildly

• When Linear Regression is used but observations are correlated (as in time series data) you will have a biased estimate of the variance

• Moreover, time-series data have a pattern, such as during peak hours, festive seasons, etc., which would most likely be treated as outliers in the linear regression analysis

Linear Regression is used to predict continuous outputs where there is a linear relationship between the features of the dataset and the output variable. It is used for regression problems where you are trying to predict something with infinite possible answers such as the price of a house.

In the case of regression, decision trees in random forest learn by splitting the training examples in a way such that the sum of squared residuals is minimized. To classify a new object based on attributes, each tree gives a classification and we say the tree “votes” for that class. The forest chooses the classification having the most votes (over all the trees in the forest) and in case of regression, it takes the average of outputs by different trees. It is useful when there are complex relationships between the features and the output variables. They also work well compared to other Algorithms when there are missing features, when there is a mix of categorical and numerical features and when there is a big difference in the scale of features.

It is difficult to tell which will perform better, it completely depends on the problem statement and the available data. Other than the points mentioned above some of the Key advantages of linear models over tree-based ones are:

• they can extrapolate (e.g. if labels are between 1-5 in train set, tree based model will never predict 10, but linear will)

• could be used for anomaly detection because of extrapolation

• interpretability (yes, tree based models have feature importance, but it’s only a proxy, weights in linear model are better)

• need less data to get good results

• Random Forest is able to discover more complex relation at the cost of time

The first point becomes clearly important in this case as we would need booking price values which might not necessarily be in the training data range.

📖Source

The objective function for linear regression where $x$ is set of input vectors and $w$ are the weights: $L(w) = E[(w^Tx-y)^2]$

Let’s assume that the noise added is Gaussian as follows: $\epsilon \sim N(0, \lambda I)$, then the new objective function is given by:$L(w) = E[(w^T(x + \epsilon)-y)^2]$.

To compute it, we simplify: $L'(w) = E[(w^T x -y + w^T\epsilon)^2]$ $L'(w) = E[(w^T x - y)^2 + 2(w^Tx-y)w^T\epsilon +w^T\epsilon \epsilon^Tw]$ $L'(w) = E[(w^T x - y)^2] + E[2(w^Tx-y)w^T\epsilon] + E[w^T\epsilon \epsilon^Tw]$

We know that the expectation for $\epsilon$ is 0 so the middle term becomes 0 and we are left with:$L'(w) = L(w) + 0 + w^TE[\epsilon \epsilon^T]w$

The last term can be simplified as: $L'(w) = L(w) + w^T\lambda Iw$

And therefore the objective function simplifies to that of L2-regularization: $L'(w) = L(w) + \lambda||w||^2$

📖Source

$L1$ and $L2$ regularization are both methods of regularization that attempt to prevent overfitting in machine learning. For a regular regression model assume the loss function is given by $L$. $L1$ adds the absolute value of the coefficients as a penalty term, whereas $L2$ adds the squared magnitude of the coefficients as a penalty term.

The loss function for the two are:

• $Loss(L_1) = L + \lambda |w_i|$

• $Loss(L_2) = L + \lambda |w_i^2|$

Where the loss function $L$ is the sum of errors squared, given by the following, where $f(x)$ is the model of interest, for example, linear regression with $p$ predictors:

$L = \sum_{i=1}^{n} (y_i - f(x_i))^2 = \sum_{i=1}^{n} (y_i - \sum_{j=1}^{p}(x_{ij}w_j) )^2 \space \text{for linear regression}$

If we run gradient descent on the weights $w$, we find that $L1$ regularization will force any weight closer to 0, irrespective of its magnitude, whereas, for the $L2$ regularization, the rate at which the weight goes towards 0 becomes slower as the rate goes towards 0. Because of this, $L1$ is more likely to “zero” out particular weights, and hence removing certain features from the model completely, leading to more sparse models.

📖Source

There are two potential cost functions here, one using the $L1$ norm and the other using the $L2$ norm. Below are two basic cost functions using an L1 and L2 norm respectively:

• $J(w) = ||Xw-y||$

• $J(w) = |Xw-y|^2$

It would be more sensible to use an $L1$ norm in this case since the $L1$ norm penalizes the outliers harder and thus gives less weight to the complete failures than the $L2$ norm does.

Additionally, it would be prudent to involve a regularization term to account for noise. If we assume that the noise added to each sensor uniformly as follows: $\epsilon \sim N(0, \lambda I)$ then using traditional $L2$ regularization, we would have the cost function: $J(w) = ||Xw-y|| + \lambda||w||^2$

However, given the fact that there are many sensors (and a broad range of how useful they are), we could instead assume that noise is added by: $\epsilon \sim N(0, \lambda D)$ where each diagonal term in the matrix D represents the error term used for each sensor (and hence penalizing certain sensors more than others). Then our final cost function is given by: $J(w) = ||Xw-y|| + \lambda w^TDw$

📖Source

A mathematical derivation like this requires us to:

• Define correct Mathematical symbols and their relationships through equations

• Recall and use the definitions of the terms like likelihood and normally distributed

• Perform Mathematical manipulation to derive the required result

Problem Setup:

A linear regression model proposes that the output $y$ is linearly dependent on the input vector $X$ by the relation,

$y = W^T X + \beta$

Where, $X$ is an $m$-dimensional vector, and $(W; \beta) = \{w_1, w_2, ..., w_m; \beta\}$ are the parameters of the model.

Next, we are give a set of training data points, consisting of

• a set of input vectors, $X = \{X_1, X_2, ..., X_n\}$. Note that every input $X_i$ is a vector, $X_i = \{X_{i1}, X_{i2}, ..., X_{im}\}$

• and a set of outputs $Y = \{y_1, y_2, ..., y_n\}$

Given the values of the parameters $W$ and $\beta$, the estimate $\hat{y}$ is given by

$\hat{y}_i = \sum_{j=1}^m X_{ij} * w_j + \beta = X_{i1} * w_1 + X_{i2} * w_2 + ... + X_{im} * w_m + \beta$.

Finally, the error term for $i^{th}$ input is simply the difference between the observed value $y$ and the estimate $\hat{y}$.

$\epsilon_i(W, \beta) = y_i - \hat{y}_i = y_i - \sum_{j=1}^m X_{ij} * w_j - \beta$

Note that the error term depends on the parameters of the model $W$ and $\beta$, and hence is denoted as $\epsilon_i(W, \beta)$.

Likelihood:

Take a look at the problem statement again. We are assuming that the error terms are normally distributed. There is an implicit assumption that all the error terms are independent of each other. (Make sure you make this assumption explicit to your interviewer).

What does it mean for the error term to be normally distributed. It means that, by definition, the probability distribution function of the $i^{th}$ error term is given by

$l(\epsilon_i|W; \beta) = \frac{1}{\sqrt{2\pi}\sigma}exp\{\frac{-\epsilon_i^2}{2\sigma^2}\}$

This probability distribution function of the $i^{th}$ input is also called its likelihood function, and also depends on the parameters of the model, $W$ and $\beta$.

Since we are assuming that the error terms are also independent, their joint probability distribution, is given by the product of their likelihood.

$l = \displaystyle \prod_{i=1}^n l(\epsilon_i|W; \beta)$ $= \displaystyle \prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma}exp\{\frac{-\epsilon_i^2}{2\sigma^2}\}$ $= \displaystyle \frac{1}{(\sqrt{2\pi}\sigma)^n} \prod_{i=1}^n exp\{\frac{-\epsilon_i^2}{2\sigma^2}\}$ $= \displaystyle \frac{1}{(\sqrt{2\pi}\sigma)^n} \prod_{i=1}^n exp\{\frac{-(y_i - \sum_{j=1}^m X_{ij} * w_j - \beta)^2}{2\sigma^2}\}$

Maximum Likelihood Estimator:

The maximum likelihood estimator seeks to maximize the likelihood function defined above. For the maximization,

• We can ignore the constant $\frac{1}{(\sqrt{2\pi}\sigma)^n}$

• We can also take the log of the likelihood function, converting the product into sum

The log likelihood function of the errors is given by

$L = log(l)$ $= \displaystyle log(\prod_{i=1}^n exp\{\frac{-(y_i - \sum_{j=1}^m X_{ij} * w_j - \beta)^2}{2\sigma^2}\})$ $= \displaystyle \sum_{i=1}^n {\frac{-(y_i - \sum_{j=1}^m X_{ij} * w_j - \beta)^2}{2\sigma^2}}$

As a final step, for the purpose of optimization, we can ignore the constant multiplier $2\sigma^2$ from the summation, giving us

$L = \displaystyle \sum_{i=1}^n -(y_i - \sum_{j=1}^m X_{ij} * w_j - \beta)^2$

But this is just the negative of the sum of squared errors!

Thus, if you want to maximize the likelihood (or log likelihood) of the errors, you better minimize the sum of squared errors of the estimates.