[{"content":"Regularization There is a tradeoff between bias (underfitting) and variance (overfitting). The optimal tradeoff requires computing the correct model complexity.\nModel Complexity : Can be a function of the parameters ($l_2$ norm) and not just the number of parameters\nRegularization : Allows us to:\nControl model complexity Prevent overfitting Regularizer Function A regularizer $R(\\theta)$, is a function which measures model complexity. It is usually nonnegative.\nIn classical methods : $R(\\theta)$ depends only on parameters $\\theta$\nIn modern methods : $R(\\theta,x,y)$ can depend on the data\n$R(\\theta)$ is added to the training loss/cost to get the regularized loss $J_\\lambda$ $$ J_\\lambda(\\theta) = \\underbrace{J(\\theta)}_\\text{training loss} + \\lambda \\underbrace{R(\\theta)}_\\text{regularizer} $$ Parameter $\\lambda$ : Determines the balance between training loss and regularization. A small $\\lambda$ serves as a \u0026ldquo;tiebreaker\u0026rdquo; for equally effective models.\nWeight Decay One of the most commonly used is $l_2$ regularization. In deep learning it is also known as weight decay, where parameter weights gradually grow smaller over time.\nThe update rule for weight decay is $$ \\theta \\quad \\leftarrow \\underbrace{(1 - \\lambda \\eta)\\ \\cdot }_\\text{decaying weights} \\theta - \\eta \\nabla J(\\theta) $$ Is equivalent to $l_2$ regularization $$ \\theta \\leftarrow \\theta - \\eta( \\underbrace{\\nabla J(\\theta)}_\\text{loss} \\quad + \\underbrace{\\lambda \\theta}_\\text{regularizer} ) $$\nImposing Inductive Bias / Structure Regularization can also impose inductive bias or structure.\nIt imposes structure that narrows the search space. This reduces the complexity of the model family and improves generalization\nBUT, if the structure is wrong it will increase bias / underfitting.\nImplicit Regularization Effect Implicit regularization of optimizers / implicit bias / algorithmic regularization is the concept that \u0026ldquo;optimizers can implicitly impose structure\u0026rdquo; on models.\nIt is a new phenomenon observed in deep learning which might be responsible for performance in the overparameterized regime.\nBackground In classical machine learning, loss functions have a unique global minima that all optimizers converge to. However in deep learning, the loss often has multiple approximate global minima.\nDifferent optimizers tend to converge to different types of global minima. The different minima have different properties that make some generalize better than others.\nExample CS229 https://cs229.stanford.edu/\nIllustration of how certain minimas generalize better\nComponents which provide Regularization Open research question, but empirically:\nLarger initial learning rates Smaller initialization Smaller batch size Momentum Types of Global Minima which Generalize well A conjecture is that stochasticity in the optimizer helps to find flatter minima, which generalize better\nCross Validation Cross validation is a method of training and testing models on different portions of data to access the performance (true generalization) of the model.\nSimple (Hold-Out) Cross Validation Split dataset $S$ into $S_\\text{train}$ (approx. 70%) and $S_\\text{test}$(approx. 30%) Train various models on $S_\\text{train}$ and test on $S_\\text{test}$ Pick model with the smallest error This methods generates better estimates of a model\u0026rsquo;s generalization / test error, because it is tested on unseen data.\nDisadvantages It \u0026ldquo;wastes\u0026rdquo; around 30% of the dataset, even if the model is subsequently retrained on the entire dataset.\nK-Fold Cross Validation Split the data into $k$ disjoint subsets Test each model on all but one of the subsets Select model with the smallest error Retrain model on entire dataset Is a method to reduce the amount of data wasted. It only holds out $\\frac{1}{k}$ of the data.\nAdvantages : More efficient at using data, it uses $\\frac{k-1}{k}$ of the data\nDisadvantage : Computationally expensive, each model is trained $k$ times\n$k=10$ is a common choice\nLeave-one-out Cross Validation Is a more extreme variant where $k = m$, the number of training examples. At every iteration, only one training example is left out.\nBayesian Regularization Frequentist Bayesian $\\theta$ unknown constant-valued $\\theta$ is a random variable Maximum likelihood estimation (MLE) Maximum a posterior (MAP) $\\arg \\max_\\theta \\prod p(y | x;\\theta)$ $\\arg \\max_\\theta \\prod p(y | x;\\theta) \\ p(\\theta)$ Previous sections treated the parameter fitting as a maximum likelihood estimation (MLE) problem. The parameters $\\theta$ are treated as an unknown constant-value. $$ \\theta_\\text{MLE} = \\arg \\max_\\theta \\prod^n_{i=1} {p(y^{(i)}\\ | x^{(i)} ;\\ \\theta)} $$ An alternative interpretation is the Bayesian view, where $\\theta$ is a random variable (RV).\nBayesian Prediction In \u0026ldquo;fully Bayesian\u0026rdquo; prediction, the prediction is the average w.r.t. the posterior $p(\\theta | S)$ over $\\theta$ $$ \\mathbb E [y | x, S] = \\int_y y\\ p(y|x,S)\\ dy $$ Where the posterior distribution on the class label is $$ p(y|x,S) = \\int_\\theta p(y|x,\\theta) \\ p(\\theta|S) \\ d\\theta $$ And the posterior distribution on the class labels is computed from Bayes rule $$ p(\\theta | S) = \\frac {\\left( \\prod p(y|x,\\theta) \\right) p(\\theta)} {\\int_\\theta \\left( \\prod p(y|x,\\theta) \\right) p(\\theta) \\ d\\theta} $$\nMaximum A Posteriori Approximation In general it is very difficult / expensive to compute the posterior distribution. There is no closed-form solution, and the high-dimension $\\theta$ makes it intractable.\nA common approximation is to use a single point estimate, the maximum a posteriori (MAP) estimate $$ \\theta_\\text{MAP} = \\arg \\max_\\theta \\prod^n_{i=1} {p(y^{(i)}\\ | x^{(i)} ;\\ \\theta)} \\ p(\\theta) $$\nChoice of Prior $p(\\theta)$ A common choice is $$ \\theta \\sim \\mathcal{N} (0, \\tau^2 I) $$ Which has a smaller norm than MLE. This reduces the chance of overfitting.\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/regularization/","summary":"Regularization There is a tradeoff between bias (underfitting) and variance (overfitting). The optimal tradeoff requires computing the correct model complexity.\nModel Complexity : Can be a function of the parameters ($l_2$ norm) and not just the number of parameters\nRegularization : Allows us to:\nControl model complexity Prevent overfitting Regularizer Function A regularizer $R(\\theta)$, is a function which measures model complexity. It is usually nonnegative.\nIn classical methods : $R(\\theta)$ depends only on parameters $\\theta$","title":"Regularization and Model Selection"},{"content":"Generalisation The ultimate goal of machine learning is to create a predictive model which performs well on unseen examples (it generalises well).\nGeneralization is a model\u0026rsquo;s performance on unseen test data, measured by test error (loss on unseen test data).\nTest Error Loss/error on test examples $(x,y)$ sampled from a test distribution $\\mathcal{D}$ $$ L(\\theta) = \\mathbb{E}_{(x,y)\\sim\\mathcal{D}} [ (y-h_\\theta(x))^2] $$ The expectation $\\mathbb E$ can be approximated by averaging many samples\nIn classical statistical learning, both training and test examples are drawn from the same distribution.\nReasons for Large Test Error There are two main reasons for large test error:\nUnderfitting / Bias Overfitting / Variance Both training and test error are large Test error is large Model family is unable to capture the true relationship of the data Model is fitting to noise which does not reflect the true relationship of the data Underfitting / Bias Model family is unable to capture the true relationship of the data\nTest error even if the model were fitted to an infinitely large training set\nOverfitting / Variance Model is fitting to noise which does not reflect the true relationship of the data\nTradeoff between Bias \u0026amp; Variance If the model is:\nToo simple : It may have large bias Too complex : It may have large variance Decomposition of Test Error Test error can be decomposed into three terms $$ \\text{Test Error} = \\underbrace{\\quad \\sigma^2 \\quad}_\\text{noise} + \\underbrace{(h^*(x) - h_{avg}(x))^2}_\\text{bias} + \\underbrace{\\text{var}(h_S(x))}_\\text{variance} $$ Which results in a convex curve with one global optimum\nCS229 https://cs229.stanford.edu/\nDouble Descent Phenomenon Empirical observation that test error can have a second descent, in the over-parameterized regime (where number of parameters is larger than number of data points)\nCS229 https://cs229.stanford.edu/\nInterestingly in many cases there is no second ascent. Meaning that larger models always lead to better performance\nComplexity Measure of Models It\u0026rsquo;s not clear that number of parameters is the best way to measure complexity of a model\nIf the norm is used instead, there is no double descent phenomenon.\nCS229 https://cs229.stanford.edu/\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/generalisation/","summary":"Generalisation The ultimate goal of machine learning is to create a predictive model which performs well on unseen examples (it generalises well).\nGeneralization is a model\u0026rsquo;s performance on unseen test data, measured by test error (loss on unseen test data).\nTest Error Loss/error on test examples $(x,y)$ sampled from a test distribution $\\mathcal{D}$ $$ L(\\theta) = \\mathbb{E}_{(x,y)\\sim\\mathcal{D}} [ (y-h_\\theta(x))^2] $$ The expectation $\\mathbb E$ can be approximated by averaging many samples","title":"Generalization"},{"content":"Reinforcement Learning Many sequential decision making and control problems are hard to provide explicit supervision for.\nInstead provide a reward function and let the learning algorithm figure out how to choose actions over time.\nSuccessful in applications such as helicopter flight, legged locomotion, network routing, marketing strategy selection, etc\nMarkov Decision Processes (MDP) Provide formalism for many RL problems\nTerminology States : $s$ Actions : $a$ State Transition Probabilities : $P_{sa}$ Discount Factor : $\\gamma \\in [0,1)$ Reward Function : $R : S \\times A \\mapsto \\mathbb{R}$ Dynamics of MDPs Start in some state $s_0$ Choose some action $a_0 \\in A$ MDP randomly transitions to successor state $s_1 \\sim P_{s_0a_0}$ Pick another action Repeat Represent as $$ s_0 \\overset{a_0}{\\rightarrow} s_1 \\overset{a_1}{\\rightarrow} s_2 \\overset{a_2}{\\rightarrow} s_3 \\overset{a_3}{\\rightarrow} \\dots $$ Total Payoff $$ R(s_0, a_0) + \\gamma R(s_1, a_1) + \\gamma^2 R(s_2, a_2) + \\dots $$ Goal : Maximise expected value of total payoff $$ E \\left[ R(s_0) + \\gamma R(s_1) + \\gamma^2 R(s_2) + \\dots \\right] $$\nPolicy Function $\\pi : S \\mapsto A$ maps states to actions\nExecute a policy by following the prescribed action\nValue Function The expected sum of discounted rewards upon starting in $s$ and taking actions according to $\\pi$ $$ V^\\pi(s) = \\mathbb{E} \\left[ R(s_0) + \\gamma R(s_1) + \\gamma^2 R(s_2) + \\dots | s_0 = s,\\pi \\right] $$\nBellman Equation Given a fixed policy $\\pi$, it\u0026rsquo;s value function $V^\\pi$ satisfies the Bellman equations $$ V^\\pi(s) = R(s) + \\gamma \\sum_{s\u0026rsquo; \\in S} P_{s\\pi(s)}(s\u0026rsquo;) V^\\pi(s\u0026rsquo;) $$ Which consists of two terms:\nImmediate reward $R(s)$ Get for starting in $s$ Expected sum of future discounted rewards Can we rewritten as $E_{s\u0026rsquo;\\sim P_{s\\pi(s)}}[V^\\pi(s\u0026rsquo;)]$ The expected sum of starting in $s\u0026rsquo;$ where $s\u0026rsquo;$ is distributed according to to $P_{s\\pi(s)}$ Can be used to efficiently solve for the value function $V^\\pi$\nWrite down one equation for every state Gives a set of $|S|$ linear equations in $|S|$ variables Optimal Value Function Best possible expected sum of discounted rewards which can be attained using any policy\n$$ V^*(s) = \\max_\\pi V^\\pi (s) $$\nWhich has it\u0026rsquo;s own version of the Bellman Equation\n$$ V^{\\ast}(s) = R(s) + \\max_{a\\in A} \\gamma \\sum_{s\u0026rsquo;\\in S}{P_{sa} (s\u0026rsquo;) V^{\\ast}(s\u0026rsquo;)} $$\nThe second term is a $\\max$ over all possible actions because that is the optimal reward.\nOptimal Policy $$ \\pi^{\\ast}(s) = \\arg \\max_{a\\in A} \\sum_{s\u0026rsquo;\\in S} P_{sa}(s\u0026rsquo;) V^*(s\u0026rsquo;) $$\nWhich gives the action that attains the $\\max$ in the optimal value function\nFor every state $s$ and policy $\\pi$ $$ V^\\ast(s) = V^{\\pi*}(s) \\geq V^\\pi(s) $$ Which says that\nThe value function for the optimal policy is equal to the optimal value function for every state $s$\nThe value function for the optimal policy is greater than or equal to every other policy\n$\\pi^*$ is the optimal policy for all states Value Iteration and Policy Iteration Two efficient algorithms for solving finite-state MDPs with known transition probabilities\nValue Iteration For each state s, initialize $V(s) := 0$ for until convergence do For every state, update $$ V(s) := R(s) + \\max_{a\\in A} \\gamma \\sum_{s\u0026rsquo;} P_{sa}(s\u0026rsquo;) V(s\u0026rsquo;) $$\nRepeatedly update estimated value function using the Bellman equation Can be:\nSynchronous Compute new values for all states and update at once Can be viewed as implementing a Bellman Backup Operator, which maps the current estimate to a new estimate Asynchronous Loop over states and update one at a time Both algorithms will cause $V$ to converge to $V^*$\nPolicy Iteration Initialise $\\pi$ randomly\nfor until convergence do\n​\tLet $V := V^\\pi$\n​\tFor each state $s$ $$ \\pi(s) := \\arg \\max_{a \\in A} \\sum_{s\u0026rsquo;} P_{sa}(s\u0026rsquo;) V(s\u0026rsquo;) $$\nRepeatedly computes value function for the current policy Update policy using current value function This is the policy that is greedy w.r.t. $\\bold V$ The value function can be updated using Bellman equations After a finite number of iterations, $V$ will converge to $V^\\ast$ and $\\pi$ to $\\pi^\\ast$ Value vs Policy Iteration No universal agreement which is better Small MDPs : Policy iteration tends to be fast Larger MDPs : Value iteration might be faster Policy iteration can reach the exact $V^*$ Value iteration will have small non-zero error Learning a Model for an MDP In many realistic problems, we do not know the state transition probabilities and rewards\nMust be estimated from data\nGiven enough trials, can derive maximum likelihood estimates for the state transition probabilities $$ P_{sa}(s\u0026rsquo;) = \\frac{num. \\text{ times } a \\text{ led to } s\u0026rsquo; } {num. \\text{ times we took } a} $$ In the case of $\\frac{0}{0}$, can set $P$ to $\\frac{1}{|S|}$ (estimate as uniform)\nEfficient Update Keep count of numerator \u0026amp; denominator\nSolving Learned MDP Can use policy/value iteration Continuous State MDPs Infinite number of states Consider settings where state space is $S = \\mathbb{R}^d$ Discretization Simplest way Discretize state space Use value/policy iteration Downsides\nFairly naive representation for $V^*$ Assumes the value function is constant over each discretization interval Piecewise constant representation isn\u0026rsquo;t good for many smooth functions Little smoothing over inputs No generalization over different grid cells Suffers from curse of dimensionality For $S = \\mathbb{R}^d$, we will have $k^d$ states Rule of Thumb\nWorks well for 1D/2D cases Can work for 3D/4D cases Rarely works above 6D Value Function Approximation Approximate $V^*$ directly Using Model/Simulator Assume we have a model/simulator for the MDP Any black-box with Input : State $s_t$, Action $a_t$ Output : Next-state $s_{t+1}$ sampled Getting the Model Using physics simulation Can use off-the-shelf physics simulation Data collected in the MDP Can be done with: Random actions Executing some policy Other way of choosing actions [?] Control law? Apply learning algorithm to predict $s_{t+1}$ as function of $s_t$ Example : Learn linear model\nusing linear regression $$ \\arg \\min_{A,B} \\sum_{i=1}^{n} \\sum_{t=0}^{T-1} \\left\\Vert s_{t+1}^{(i)} - (A s_t^{(i)} + Ba_t^{(i)}) \\right\\Vert^2_2 $$ Can build either a deterministic $$ s_{t+1} = As_t + Ba_t $$ or stochastic model $$ s_{t+1} = As_t + Ba_t + \\epsilon_t $$ with $\\epsilon_t$ being a noise term usually modelled as $\\epsilon_t \\sim \\mathcal{N}(0,\\Sigma)$\nor non-linear function $$ s_{t+1} = A\\phi_s(s_t) + B\\phi_a(a_t) $$ Eg: Locally weighted linear regression\nFitted Value Iteration Algorithm for approximating the value function of a continuous state MDP Assume state space $S = \\mathcal{R}^d$ But action space $A$ is small and discrete Perform value iteration update $$ \\begin{aligned} V(s) :\u0026amp;= R(s) + \\gamma \\max_a \\int_{s\u0026rsquo;} P_{sa}(s\u0026rsquo;) V(s\u0026rsquo;) ds\u0026rsquo; \\cr \u0026amp;= R(s) + \\gamma \\max_a E_{s\u0026rsquo; \\sim P_{sa}} [V(s\u0026rsquo;)] \\end{aligned} $$\nMain difference is the $\\int$ instead of $\\sum$ Approximately update over a finite sample of states $s^{(1)}, \\dots, s^{(n)}$ Use supervised learning algorithm, linear regression to approximate value function as linear/nonlinear function of the states $$ V(s) = \\theta^T \\phi(s) $$\nAlgorithm\nCompute $y^{(i)}$, the approximation to $R(s) + \\gamma \\max_a E_{s\u0026rsquo; \\sim P_{sa}} [V(s\u0026rsquo;)]$ (RHS)\nUse supervised learning to get $V(s) = y^{(i)}$\nRepeat\nConvergence Cannot be proved to converge But in practice it often does, and works well for many problems Deterministic Simplification Can set $k=1$ Because expectation of a deterministic distribution requires 1 sample Policy Outputs $V$, an approximation of $V^*$ Implicitly defines policy When in state $s$, choose $$ \\arg \\max_a E_{s\u0026rsquo;\\sim P_{sa}} [V(s\u0026rsquo;)] $$\nProcess for computing/approximating this is similar to the fitted value iteration algorithm ","permalink":"https://andrelimzs.github.io/posts/machine-learning/reinforcement-learning/","summary":"Reinforcement Learning Many sequential decision making and control problems are hard to provide explicit supervision for.\nInstead provide a reward function and let the learning algorithm figure out how to choose actions over time.\nSuccessful in applications such as helicopter flight, legged locomotion, network routing, marketing strategy selection, etc\nMarkov Decision Processes (MDP) Provide formalism for many RL problems\nTerminology States : $s$ Actions : $a$ State Transition Probabilities : $P_{sa}$ Discount Factor : $\\gamma \\in [0,1)$ Reward Function : $R : S \\times A \\mapsto \\mathbb{R}$ Dynamics of MDPs Start in some state $s_0$ Choose some action $a_0 \\in A$ MDP randomly transitions to successor state $s_1 \\sim P_{s_0a_0}$ Pick another action Repeat Represent as $$ s_0 \\overset{a_0}{\\rightarrow} s_1 \\overset{a_1}{\\rightarrow} s_2 \\overset{a_2}{\\rightarrow} s_3 \\overset{a_3}{\\rightarrow} \\dots $$ Total Payoff $$ R(s_0, a_0) + \\gamma R(s_1, a_1) + \\gamma^2 R(s_2, a_2) + \\dots $$ Goal : Maximise expected value of total payoff $$ E \\left[ R(s_0) + \\gamma R(s_1) + \\gamma^2 R(s_2) + \\dots \\right] $$","title":"Reinforcement Learning"},{"content":"Deep Learning Supervised with Nonlinear Models Supervised learning is\nPredict $y$ from input $x$ Suppose model/hypothesis is $h_\\theta(x)$ Previous methods have considered\nLinear regression : $h_\\theta(x) = \\theta^Tx$ Kernel method : $h_\\theta(x) = \\theta^T \\phi(x)$ Both are linear in $\\theta$\nNow consider models that are nonlinear in both\nParameters : $\\theta$ Inputs : $x$ The most common of which is the neural network\nCost/Loss Function Define least-squares cost for one sample $$ J^{(i)}(\\theta) = \\frac{1}{2} ( h_\\theta(x^{(i)}) - y^{(i)})^2 $$ and mean-square cost for the dataset $$ J(\\theta) = \\frac{1}{n} \\sum_{i=1}^{n} J^{(i)}(\\theta) $$\nOptimisers SGD or it\u0026rsquo;s variants is commonly used $$ \\theta := \\theta - \\alpha \\nabla_\\theta J(\\theta) $$ with $\\alpha \u0026gt; 0$\nHardware parallelisation means that it\u0026rsquo;s often faster to compute the gradient of $B$ examples simultaneously\n$\\Rightarrow$ Mini-batch SGD is most commonly used\nNeural Networks Refers to any type of nonlinear model/parameterisation $h_\\theta(x)$ that involves matrix multiplications and other entrywise nonlinear operations\nActivation Function A (1D) nonlinear function that maps $\\mathbb{R}$ to $\\mathbb{R}$\nCommon ones include\nReLU : Rectified Linear Unit Single Neuron $$ h_\\theta(x) = ReLU(w^Tx + b) $$\nInput : $x \\in \\mathbb{R}^d$ Weights : $w \\in \\mathbb{R}^d$ Bias : $b \\in \\mathbb{R}$ Stacking Neurons Enabels more complex networks\n[Example] Two-layer Fully-connected Network\nFor a hidden layer with three neurons $$ a_1 = ReLU(w_1^T x + b_1) \\\\ a_2 = ReLU(w_2^T x + b_2) \\\\ a_3 = ReLU(w_3^T x + b_3) $$ Or equivalently $$ \\begin{aligned} \u0026amp;z_i = (w_i^{[1]})^T x + b_i^{[1]} \\quad \\forall i \\in [1,\\dots,m] \\cr \u0026amp; a_i = ReLU(z_i) \\cr \u0026amp; a = [ a_1,\\ \\dots,\\ a_m ]^T \\cr \u0026amp; h_\\theta(x) = (w^{[2]})^T a + b^{[2]} \\end{aligned} $$\nVectorisation Simplify expression using more matrix/vector notation Important in speeding up neural networks Takes advantage of optimised linear algebra packages such as LAPACK/BLAS Leverages GPU parallelism Concatentae weights/inputs/outputs as $$ W^{[i]} = \\begin{bmatrix} -\\ w^{[i]\\top}_1 \\ - \\cr \\vdots \\cr -\\ w^{[i]\\top}_m \\ - \\end{bmatrix} \\in \\mathbb{R}^{m\\times d} $$ To get $$ \\underbrace{ \\begin{bmatrix} z^{[i]}_1 \\cr \\vdots \\cr z^{[i]}_m \\end{bmatrix} }_{z \\in \\mathbb{R}^{m\\times 1}} = \\underbrace{ \\begin{bmatrix} -\\ w^{[i]\\top}_1 \\ - \\cr \\vdots \\cr -\\ w^{[i]\\top}_m \\ - \\end{bmatrix} }_{W^{[1]} \\in \\mathbb{R}^{m\\times d}} \\underbrace{ \\begin{bmatrix} x_1 \\cr \\vdots \\cr x_d \\end{bmatrix} }_{x \\in \\mathbb{R}^{d\\times 1}} + \\underbrace{ \\begin{bmatrix} b^{[1]}_1 \\cr \\vdots \\cr b^{[1]}_m \\end{bmatrix} }_{b^{[1]} \\in \\mathbb{R}^{m\\times 1}} $$ Or equivalently $$ \\begin{aligned} \u0026amp;z = W^{[1]}x + b^{[1]} \\cr \u0026amp;a = ReLU(W^{[1]} + b^{[1]}) \\cr \u0026amp;h_\\theta(x) = W^{[2]} a + b^{[2]} \\end{aligned} $$\n$\\theta$ consists of weights $W^{[i]}$ and biases $b^{[i]}$\nEach layer consists of the pair $(W^{[i]}, b^{[i]})$\nMulti-Layer Network Can stack more layers to get a deeper fully-connected neural network If you let $a^{[0]} = x$ and $a^{[r]} = h_\\theta(x)$ $$ a^{[k]} = \\text{ReLU}\\left( W^{[k]} a^{[k-1]} + b^{[k]} \\right), \\quad \\forall k=1,\\dots,r-1 $$\nConnection to Kernel Method Deep learning is a method to automatically learn the best feature map (or representation) $$ h_\\theta(x) = W^{[r]} \\phi_\\beta(x) + b^{[r]} $$ With $\\beta: a^{[r-1]} = \\phi_\\beta(x)$ as the feature map\nThe neural network is therefore a linear regression over the learnt (nonlinear) feature map $\\phi_\\beta$\nBackpropagation Aka auto-differentiation\nCompute the gradient of the loss $\\nabla J^{(j)} (\\theta)$ efficiently\nTheorem (Informally)\nSuppose a differentiable circuit of size $N$ computes a real-valued function $f:\\mathbb R^l \\rightarrow \\mathbb R$. The gradient $\\nabla f$ can be computed in $O(N)$ time by a circuit of size $O(N)$\nChain Rule For variables $\\theta_1, \\dots, \\theta_p$ and intermediate variables $g_1, \\dots, g_k$ $$ \\begin{aligned} g_j = g_j(\\theta_1, \\dots, \\theta_p) \\cr J = J(g_1, \\dots, g_k) \\end{aligned} $$ The chain rule states $$ \\frac{\\partial J}{\\partial \\theta_i} = \\sum^{k}_{j=1} \\frac{\\partial J}{\\partial g_j} \\frac{\\partial g_j}{\\partial \\theta_i} $$\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/deep-learning/","summary":"Deep Learning Supervised with Nonlinear Models Supervised learning is\nPredict $y$ from input $x$ Suppose model/hypothesis is $h_\\theta(x)$ Previous methods have considered\nLinear regression : $h_\\theta(x) = \\theta^Tx$ Kernel method : $h_\\theta(x) = \\theta^T \\phi(x)$ Both are linear in $\\theta$\nNow consider models that are nonlinear in both\nParameters : $\\theta$ Inputs : $x$ The most common of which is the neural network\nCost/Loss Function Define least-squares cost for one sample $$ J^{(i)}(\\theta) = \\frac{1}{2} ( h_\\theta(x^{(i)}) - y^{(i)})^2 $$ and mean-square cost for the dataset $$ J(\\theta) = \\frac{1}{n} \\sum_{i=1}^{n} J^{(i)}(\\theta) $$","title":"Deep Learning"},{"content":"Support Vector Machines The support vector machine (SVM) is a supervised learning method that finds the optimal margin for either regression or classification.\nFor the linear classifier $$ h_{w,b}(x) = g(w^T x + b) $$ with $y \\in {-1, 1}$\nMargins Margins represent the idea of how confident and correct a prediction is.\nGeometric Margin, $\\gamma$ The geometric margin is the Euclidean distance from a sample to the decision boundary, defined as\n$$ \\gamma^{(i)} = y^{(i)} \\left( \\left( \\frac{w}{||w||} \\right)^T x^{(i)} + \\frac{b}{||w||} \\right) $$\nThe geometric margin is invariant to scaling of the parameters $w$ and $b$.\nWhen $||w|| = 1$, the geometric margin equals the functional margin.\nFunctional Margin, $\\hat \\gamma$ The functional margin is defined as $$ \\hat{\\gamma}^{(i)} = y^{(i)} (w^T x^{(i)} + b) $$\nOr $$ \\hat{\\gamma} = ||w|| \\gamma $$ The functional margin can be made arbitrarily large by scaling $w$ and $b$ without changing the decision boundary.\nTraining Set Margin Given a training set $S = { x^{(i)}, y^{(i)}; i=1,\\dots,n }$ the margins are defined as the smallest of the individual training examples $$ \\hat{\\gamma} = \\min_{i=1,\\dots,n} \\hat\\gamma^{(i)} \\quad \\text{or} \\quad \\gamma = \\min_{i=1,\\dots,n} \\gamma^{(i)} $$\nOptimal Margin Classifier Assume that the training set is linearly separable (can be separated with a hyperplane), the problem can be formulated as: $$ \\begin{aligned} \u0026amp;\u0026amp;\\max_{\\gamma,w,b} \u0026amp;\u0026amp;\u0026amp;\\gamma \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp;y^{(i)} (w^T x^{(i)} + b) \\geq \\gamma, \\quad i = 1, \\dots, n \\cr \u0026amp;\u0026amp;\u0026amp;\u0026amp;\u0026amp; ||w|| = 1 \\end{aligned} $$ However $||w||=1$ is a non-convex constraint, and cannot be easily solved. The problem can be transformed using $\\gamma = \\frac{\\hat{\\gamma}}{||w||}$\n$$ \\begin{aligned} \u0026amp;\u0026amp;\\max_{\\gamma,w,b} \u0026amp;\u0026amp;\u0026amp; \\frac{\\hat\\gamma}{||w||} \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; y^{(i)} (w^T x^{(i)} + b) \\geq \\hat\\gamma, \\quad i = 1, \\dots, n \\end{aligned} $$ Because the absolute magnitude of $\\hat \\gamma$ doesn\u0026rsquo;t matter, we can set $\\hat\\gamma = 1$ to get $$ \\max_{w,b} \\quad \\frac{1}{||w||} $$ which is equivalent to $$ \\min_{w,b} \\quad ||w||^2 $$ To finally get $$ \\begin{aligned} \u0026amp;\u0026amp;\\min_{w,b} \u0026amp;\u0026amp;\u0026amp; \\frac{1}{2} ||w||^2 \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; y^{(i)} (w^T x^{(i)} + b) \\geq \\hat\\gamma, \\quad i = 1, \\dots, n \\end{aligned} $$ Which is a quadratic programming (QP) problem.\nLagrange Duality Useful in solving constrained optimization problems\nConsider a problem of the form $$ \\begin{aligned} \u0026amp;\u0026amp;\\min_{w} \u0026amp;\u0026amp;\u0026amp; f(w) \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; h_i(w) = 0, \\quad i=1,\\dots,l \\end{aligned} $$ Define the Lagrangian $$ \\mathcal{L}(w,b) = f(w) + \\sum_{i=1}^l \\beta_i h_i(w) $$ With Lagrangian multipliers $\\beta_i$\nAnd we can solve $w$ and $b$ by setting the partial derivatives of $\\mathcal L$ to zero $$ \\frac{\\partial \\mathcal L}{\\partial w_i}=0, \\quad \\frac{\\partial \\mathcal L}{\\partial \\beta_i}=0 $$\nConstrained Optimization The method can be generalized to include inequality constraints.\nFor the primal optimization problem $$ \\begin{aligned} \u0026amp;\u0026amp;\\min_{w} \u0026amp;\u0026amp;\u0026amp; f(w) \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; g_i(w) \\leq 0, \\quad i = 1, \\dots, k \\cr \u0026amp;\u0026amp; \u0026amp;\u0026amp;\u0026amp; h_i(w) = 0, \\quad i = 1, \\dots, l \\end{aligned} $$ Define the generalized Lagrangian $$ \\mathcal{L} (w,\\alpha,\\beta) = f(w) + \\sum_{i=1}^k{\\alpha_i g_i(w)} + \\sum_{i=1}^l{\\beta_i h_i(w)} $$ With Lagrange multipliers $\\alpha, \\beta$\nDefine $\\theta_\\mathcal{P}$ $$ \\theta_\\mathcal{P}(w) = \\max_{\\alpha,\\beta:\\alpha_i \\geq 0} \\mathcal L(w,\\alpha,\\beta) $$ With values $$ \\theta_\\mathcal{P} = \\left\\{ \\begin{aligned} f\u0026amp;(w) \u0026amp;\u0026amp; \\text{if w satisfies primal constraints} \\cr \u0026amp;\\infty \u0026amp;\u0026amp; \\text{otherwise} \\end{aligned} \\right. $$ Therefore the problem $$ \\min_w \\theta_\\mathcal{P}(w) = \\min_w \\max_{\\alpha,\\beta:\\alpha_i\\geq0} \\mathcal{L}(w,\\alpha,\\beta) $$ is the same as the original problems\nDefine the optimal value of the objective $p*$ $$ p* = \\min_w \\theta_\\mathcal{P}(w) $$\nDefine $$ \\theta_\\mathcal{D}(\\alpha,\\beta) = \\min_{w} \\mathcal L(w,\\alpha,\\beta) $$\nPose the dual optimization problem $$ \\max_{\\alpha,\\beta:\\alpha_i\\geq0} \\theta_\\mathcal{D}(\\alpha,\\beta) = \\max_{\\alpha,\\beta:\\alpha_i\\geq0} \\min_{w} \\mathcal{L}(w,\\alpha,\\beta) $$ Which is the same as the primal problem, with the order of $\\max$ and $\\min$ swapped\nThe primal and dual problems are related via $$ d^* = \\max_{\\alpha,\\beta:\\alpha_i\\geq0} \\min_{w} \\mathcal L(w,\\alpha,\\beta) \\leq \\min_{w} \\max_{\\alpha,\\beta:\\alpha_i\\geq0} \\mathcal L(w,\\alpha,\\beta) = p^* $$ Under Karush-Kuhn-Tucker (KKT) conditions $$ d^* = p^* $$ and this can be used to solve the primal problem\nOptimal Margin Classifiers Dual Form For the (primal) optimization problem $$ \\begin{aligned} \u0026amp;\u0026amp;\\min_{w,b} \u0026amp;\u0026amp;\u0026amp; \\frac{1}{2} ||w||^2 \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; y^{(i)} (w^T x^{(i)} + b) \\geq 1, \\quad i = 1, \\dots, n \\end{aligned} $$ Rewrite the constraints as $$ g_i(w) = -y^{(i)} (w^T x^{(i)} + b) + 1 \\leq 0 $$ With one constraint per training example\nConstruct the Lagrangian $$ \\mathcal L(w,\\beta,\\alpha) = \\frac{1}{2} ||w||^2 - \\sum_{i=1}^{n} \\alpha_i \\left[ y^{(i)} (w^T x^{(i)} + b) - 1 \\right] $$ Find the dual form\nFirst minimize $\\mathcal L(w,\\beta,\\alpha)$ w.r.t $w$ $$ \\nabla_w \\mathcal L(w,\\beta,\\alpha) = w - \\sum_{i=1}^{n} \\alpha_i y^{(i)} x^{(i)} = 0 $$ Which implies $$ w = \\sum_{i=1}^{n} \\alpha_i y^{(i)} x^{(i)} $$ Then minimize $\\mathcal L(w,\\beta,\\alpha)$ w.r.t $b$ $$ \\frac{\\partial}{\\partial\\beta} \\mathcal L(w,\\beta,\\alpha) = \\sum_{i=1}^{n} \\alpha_i y^{(i)} = 0 $$ Combine to obtain the dual optimization problem $$ \\begin{aligned} \u0026amp;\u0026amp;\\max_{\\alpha} \u0026amp;\u0026amp;\u0026amp; W(\\alpha) = \\sum_{i=1}^{n} \\alpha_i - \\frac{1}{2} \\sum_{i,j=1}^{n} y^{(i)} y^{(j)} \\alpha_i \\alpha_j \\langle x^{(i)} , x^{(j)} \\rangle \\cr \u0026amp;\u0026amp;\\text{s.t.} \u0026amp;\u0026amp;\u0026amp; \\alpha_i \\geq 0, \\quad i = 1, \\dots, n \\cr \u0026amp;\u0026amp; \u0026amp;\u0026amp;\u0026amp; \\sum_{i=1}^{n} \\alpha_i y^{(i)} = 0 \\end{aligned} $$ The original problem satisfies the KKT conditions and therefore we can solve this dual problem in lieu of the primal problem.\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/support-vector-machines/","summary":"Support Vector Machines The support vector machine (SVM) is a supervised learning method that finds the optimal margin for either regression or classification.\nFor the linear classifier $$ h_{w,b}(x) = g(w^T x + b) $$ with $y \\in {-1, 1}$\nMargins Margins represent the idea of how confident and correct a prediction is.\nGeometric Margin, $\\gamma$ The geometric margin is the Euclidean distance from a sample to the decision boundary, defined as","title":"Support Vector Machine (SVM)"},{"content":"Kernel Methods Kernel methods are an efficient way to perform nonlinear regression or classification, by calculating the dot product instead of the entire feature map.\nFeature Maps A feature map $\\phi$ is a function that maps input attributes to some new (nonlinear) feature variables.\nLMS with Features First lets define:\nInput : $x \\in \\mathbb R ^ d$ Features : $\\phi(x) \\in \\mathbb R^{p}$ Weights : $\\theta \\in \\mathbb R^d$ Modify gradient descent for ordinary least squares problem $$ \\theta := \\theta + \\alpha \\sum_{i=1}^{n}{(y^{(i)} - \\theta^T x^{(i)})\\ x^{(i)}} $$ With a feature map $\\phi : \\mathbb R^d \\rightarrow \\mathbb R^p$ that maps $x$ to $\\phi(x)$ $$ \\theta := \\theta + \\alpha \\sum_{i=1}^{n}{(y^{(i)} - \\theta^T \\underbrace{\\phi(x^{(i)})} )\\ \\underbrace{\\phi(x^{(i)}})} $$ Which will have SGD update rule $$ \\theta := \\theta + \\alpha (y^{(i)} - \\theta^T \\phi(x^{(i)}))\\ \\phi(x^{(i)}) $$\nLMS with Kernel Trick The gradient descent update becomes computationally expensive when the feature map $\\phi$ is high dimensional because the size of $\\phi(x)$ grows roughly on the order of $O(n^\\text{dim})$.\nThe kernel trick depends on the fact that the parameter vector $\\theta$ is a linear combination of the features $\\phi(x^{(i)})$. For the next section I will switch to matrix notation to make it clearer. $$ \\underbrace{ \\theta }_{ \\mathbb R^{d} } = \\underbrace{ \\Phi(x) }_{ \\mathbb R^{d \\times n} } \\underbrace{ \\beta }_{ \\mathbb R^{n} } $$\n$\\Phi(x) = \\begin{bmatrix} \\phi(x)^{(1)} \u0026amp; \\dots \u0026amp; \\phi(x)^{(n)}\\end{bmatrix} \\in \\mathbb R^{d \\times n}$ Which expanded is $$ \\begin{bmatrix} \\theta_1 \\cr \\vdots \\cr \\theta_n \\end{bmatrix} = \\begin{bmatrix} \\phi(x^{(1)})_1\\ \\beta_1 + \\dots + \\phi(x^{(n)})_1\\ \\beta_n \\cr \\vdots \\cr \\phi(x^{(1)})_n\\ \\beta_1 + \\dots + \\phi(x^{(n)})_n\\ \\beta_n \\end{bmatrix} $$ Next, the original update equation $$ \\underbrace{\\theta}_{\\mathbb R^d} := \\theta + \\alpha \\ \\underbrace{\\Phi(x)}_{\\mathbb R^{d \\times n}} \\underbrace{\\left( Y^T - \\Phi(x)^T \\theta \\right)}_{\\mathbb R^{n}} $$\n$Y = \\begin{bmatrix} y^{(1)} \u0026amp; \\dots \u0026amp; y^{(n)}\\end{bmatrix} \\in \\mathbb R^{1 \\times n}$ $\\Phi(x) = \\begin{bmatrix} \\phi(x)^{(1)} \u0026amp; \\dots \u0026amp; \\phi(x)^{(n)}\\end{bmatrix} \\in \\mathbb R^{d \\times n}$ can be transformed to coordinates of $\\beta$ $$ \\beta := \\beta + \\alpha( Y^T - \\Phi(x)^T \\theta ) $$ And finally substituted with $\\theta = \\Phi(x) \\beta$ $$ \\beta := \\beta + \\alpha( Y^T - \\underbrace{\\Phi(x)^T \\Phi(x)}_\\text{inner product} \\ \\beta ) $$ Where $\\Phi(x)^T \\Phi(x)$ is also known as the dot/inner product and often written as $$ \\Phi(x)^T \\Phi(x) = \\langle \\Phi(x),\\Phi(x) \\rangle $$\nWhy Is It Faster The inner product $\\langle \\phi(x^{(j)}),\\phi(x^{(i)}) \\rangle$ can be precomputed The inner product can be computed without computing the feature map explicitly $$ \\langle x,z \\rangle = 1 + \\langle x,z \\rangle + \\langle x,z \\rangle^2 + \\langle x,z \\rangle^3 $$\nWhich takes $O(d)$ for $\\langle x,z \\rangle$ then constant operations for the higher powers\nKernels Function that maps $\\chi \\times \\chi \\rightarrow \\mathbb R$ $$ K(x,z) \\triangleq \\langle \\phi(x), \\phi(z) \\rangle $$\nProcedure Compute kernel $$ K(x^{(i)},x^{(j)}) \\triangleq \\langle \\phi(x^{(i)}), \\phi(x^{(j)}) \\rangle $$\nIterate through samples and update $$ \\beta_i := \\beta_i + \\alpha \\left( y^{(i)} - \\sum_{j=1}^n \\beta_j K(x^{(i)},x^{(j)}) \\right), \\quad \\forall i \\in {1,\\dots n} $$\nOr equivalently $$ \\beta := \\beta + \\alpha( Y^T - K\\beta ) $$\nThe representation $\\beta$ is sufficient to compute the prediction $\\theta^T \\phi(x)$ $$ \\theta^T \\phi(x) = \\sum_{i=1}^{n} \\beta_i K(x^{(i)},x) $$\nProperties of Kernels Necessary and Sufficient Condition Suppose $K$ is a valid kernel corresponding to some feature map $\\phi$\nConsider $n$ points ${ x^{(1)}, \\dots, x^{(n)} }$\nDefine a square $n\\times n$ kernel matrix $K$ with the $(i,j)$-entry given by $$ K_{ij} = K(x^{(i)},y^{(j)}) $$ If $K$ is a valid matrix:\n$K_{ij} = K_{ji}$ (Symmetric)\n$z^T K Z \\geq 0$ (PSD)\nWhich means that kernel matrix $K \\in \\mathbb{R}^{n \\times n}$ is positive semidefinite\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/kernel-methods/","summary":"Kernel Methods Kernel methods are an efficient way to perform nonlinear regression or classification, by calculating the dot product instead of the entire feature map.\nFeature Maps A feature map $\\phi$ is a function that maps input attributes to some new (nonlinear) feature variables.\nLMS with Features First lets define:\nInput : $x \\in \\mathbb R ^ d$ Features : $\\phi(x) \\in \\mathbb R^{p}$ Weights : $\\theta \\in \\mathbb R^d$ Modify gradient descent for ordinary least squares problem $$ \\theta := \\theta + \\alpha \\sum_{i=1}^{n}{(y^{(i)} - \\theta^T x^{(i)})\\ x^{(i)}} $$ With a feature map $\\phi : \\mathbb R^d \\rightarrow \\mathbb R^p$ that maps $x$ to $\\phi(x)$ $$ \\theta := \\theta + \\alpha \\sum_{i=1}^{n}{(y^{(i)} - \\theta^T \\underbrace{\\phi(x^{(i)})} )\\ \\underbrace{\\phi(x^{(i)}})} $$ Which will have SGD update rule $$ \\theta := \\theta + \\alpha (y^{(i)} - \\theta^T \\phi(x^{(i)}))\\ \\phi(x^{(i)}) $$","title":"Kernel Methods"},{"content":"Generative Learning Generative learning is a different approach to learning as opposed to discriminative learning. It tries to model $p(x|y)$ and $p(y)$ instead of learning $p(y|x)$ directly.\n$p(x|y)$ : Distribution of the target\u0026rsquo;s features $p(y)$ : Class priors It uses Bayes rule to calculate posterior distribution $p(y|x)$ $$ p(y|x) = \\frac{p(x|y) p(y)}{p(x)} $$ And can be simplified when using for prediction because the denominator is constant and doesn\u0026rsquo;t matter $$ \\arg \\max_y P(y|x) = \\arg \\max_y p(x|y) p(y) $$\nMultivariate Normal Distribution The multivariate normal distribution in $d$-dimensions is parameterised by:\nMean Vector : $\\mu \\in \\mathbb{R}^d$ Covariance Matrix : $\\Sigma \\in \\mathbb{R}^{d\\times d}$ Where $\\Sigma$ is symmetric and positive semi-definite (PSD)\nThe density of $\\mathcal{N} (\\mu, \\Sigma)$ is $$ p(x; \\mu, \\Sigma) = \\frac{1}{(2\\pi)^{d/2}|\\Sigma|^{1/2}} \\exp \\left( -\\frac{1}{2} (x-\\mu)^T \\Sigma^{-1} (x-\\mu) \\right) $$ Where $|\\Sigma|$ is the determinant of $\\Sigma$\nMean Is given by $\\mu$ $$ \\mathbb{E}[X] = \\int_x{ x\\ p(x; \\mu,\\Sigma)\\ dx } = \\mu $$\nCovariance Is defined as $$ Cov(X) = \\mathbb{E} \\left[ \\ (X-\\mathbb{E}[X])(X-\\mathbb{E}[X])^T \\ \\right] $$\n$$ Cov(X) = \\mathbb{E}[XX^T] - (\\mathbb{E}[X])(\\mathbb{E}[X])^T $$\nAs covariance increases, the distribution becomes more spread out. Off-diagonal terms skew the distribution.\nThe standard normal distribution has zero mean and identity covariance.\nGaussian Discriminant Analysis (GDA) For a classification problem with continuous input features, the GDA model assumes that $p(x|y)$ is distributed according to a multivariate normal distribution $$ \\begin{aligned} y \u0026amp;\\sim \\text{Bernoulli}(\\phi) \\cr x|y=0 \u0026amp;\\sim \\mathcal{N}(\\mu_0, \\Sigma) \\cr x|y=1 \u0026amp;\\sim \\mathcal{N}(\\mu_1, \\Sigma) \\end{aligned} $$ Which expands to $$ \\begin{aligned} p(y) \u0026amp;\\sim \\phi^y (1-\\phi)^{1-y} \\cr p(x|y=0) \u0026amp;\\sim \\frac{1}{(2\\pi)^{d/2}|\\Sigma|^{1/2}} \\exp \\left( -\\frac{1}{2} (x-\\mu_0)^T \\Sigma^{-1} (x-\\mu_0) \\right) \\cr p(x|y=1) \u0026amp;\\sim \\frac{1}{(2\\pi)^{d/2}|\\Sigma|^{1/2}} \\exp \\left( -\\frac{1}{2} (x-\\mu_1)^T \\Sigma^{-1} (x-\\mu_1) \\right) \\cr \\end{aligned} $$ This model usually sets $\\Sigma_0 = \\Sigma_1 = \\Sigma$\nThe log-likelihood is $$ \\begin{aligned} l(\\phi, \\mu_0, \\mu_1, \\Sigma) \u0026amp;= log \\prod_{i=1}^{n} p(x^{(i)}, y^{(i)}; \\phi, \\mu_0, \\mu_1, \\Sigma) \\cr \u0026amp;= \\log \\prod_{i=1}^{n} p(x|y; \\mu, \\mu, \\Sigma)\\ p(y;\\phi) \\end{aligned} $$ And the maximum likelihood estimates of the parameters can be found by maximising $l$ w.r.t the parameters $$ \\phi = \\frac{1}{n} \\sum_{i=1}^{n} 1{ y^{(i)}=1 } $$\n$$ \\mu_0 = \\frac{\\sum_{i=1}^n 1{ y^{(i)}=0 } x^{(i)}} {\\sum_{i=1}^n 1{ y^{(i)}=0 }} $$\n$$ \\mu_1 = \\frac{\\sum_{i=1}^n 1{ y^{(i)}=1 } x^{(i)}} {\\sum_{i=1}^n 1{ y^{(i)}=1 }} $$\n$$ \\Sigma = \\frac{1}{n} \\sum_{i=1}^{n} (x^{(i)} - \\mu_{y^{(i)}}) (x^{(i)} - \\mu_{y^{(i)}})^T $$\nGraphically, the algorithm is learning the two gaussian distributions\nThe decision boundary is separating hyperplane between the two distributions\nGDA and Logistic Regression Similarities Expressing $y$ as a function of $x$ $$ p(y=1|x; \\phi, \\Sigma, \\mu_0, \\mu_1) = \\frac{1}{1 + \\exp(-\\theta^Tx)} $$ Has the exact same form as logistic regression\nDifferences GDA and logistic regression will generally give different decision boundaries because:\n$p(x|y)$ is Gaussian implies $p(y|x)$ is Logistic But $p(y|x)$ is Logistic does not imply that $p(x|y)$ is Gaussian Conclusion GDA makes stronger modelling assumptions than logistic regression\nIf assumptions are correct, GDA will find better fits If $p(x|y)$ is gaussian with shared covariance $\\Sigma$, GDA is asymptotically efficient: As $n \\rightarrow \\infty$, no algorithm is strictly better than GDA Comparison GDA Logistic Stronger modelling assumptions Less sensitive to modelling errors More data efficient More robust [*] In practise logistic regression is used more often than GDA\nNaive Bayes The naive bayes model can be used for classification with discrete-valued input features. (Similar to GDA but with discrete-valued $x$)\nProcedure:\nEncode set of features into a vocabulary. Dimension of $x$ equals size of vocabulary\nBut the number of parameters grows with $\\large 2^\\text{size of vocabulary}$ and quickly becomes too large for an explicit representation\nNaive Bayes Assumption To simplify the model we can make a very strong assumption, the naive bayes assumption:\nAssume that the $x$\u0026rsquo;s are conditially independent given $y$\nExample If an email is spam ($y=1$) and word $x_1$ is apple and word $x_2$ is car\nKnowing that $y=1$ and $x_1 = apple$ gives no information about $x_2 = car$ $$ p(x_2 | y) = p(x_2 | y, x_1) $$\n[*] This does not say that $x_1$ and $x_2$ are independent\n$$ \\begin{aligned} p(\u0026amp;x_1, \\dots, x_{5000} | y) \\cr \u0026amp;= p(x_1|y) p(x_1|y,x_1) \\dots p(x_{5000}|y,x_1\\dots x_{4999}) \\cr \\end{aligned} $$ Using the naive bayes assumption the probability can be decomposed into $$ \\begin{aligned} p(\u0026amp;x_1, \\dots, x_{5000} | y) \\vphantom{\\prod} \\cr \u0026amp;= p(x_1|y) p(x_1|y) \\dots p(x_{5000}|y) \\vphantom{\\prod} \\cr \u0026amp;= \\prod_{j=1}^d p(x_j | y) \\end{aligned} $$ The NB assumption is extremely strong, but the algorithm works well on many real life problems.\nParameterisation The joint likelihood is $$ \\mathcal{L}(\\phi_y, \\phi_{j|y=0}, \\phi_{j|y=1}) = \\prod_{i=1}^{n} p(x^{(i)}, y^{(i)}) $$ Maximise w.r.t $\\phi_y$, $\\phi_{j|y=0}$ and $\\phi_{j|y=1}$ $$ \\begin{aligned} \\phi_{j|y=1} \u0026amp;= \\frac{\\sum_{i=1}^n 1{ x_k^{(i)}=1 \\wedge y^{(i)}=1 }} {\\sum_{i=1}^n 1{ y^{(i)}=1 }} \\cr \\phi_{j|y=0} \u0026amp;= \\frac{\\sum_{i=1}^n 1{ x_k^{(i)}=1 \\wedge y^{(i)}=0 }} {\\sum_{i=1}^n 1{ y^{(i)}=0 }} \\cr \\phi_{y} \u0026amp;= \\frac{\\sum_{i=1}^n 1{ y^{(i)}=1 }}{n} \\end{aligned} $$ To make a prediction, find the highest possiblity from $$ p(y=1|x) = \\frac{p(x|y=1) p(y=1)}{p(x)} $$\nLaplace Smoothing The first time the Naive Bayes algorithm encounters a new feature, it cannot estimate the probability $$ \\begin{aligned} p(y=1|x) \u0026amp;= \\frac { \\prod_{j=1}^{d} p(x_j|y=1) p(y=1) } { \\prod_{j=1}^{d} p(x_j|y=1) p(y=1) + \\prod_{j=1}^{d} p(x_j|y=0) p(y=0) } \\cr p(y=1|x) \u0026amp;= \\frac{0}{0} \\end{aligned} $$\nBecause $\\prod p(x_j|y)$ includes $p(x_{new}|y)$ which is $0$ and therefore it always obtains $0/0$. It is a bad idea to estimate a previously unseen event to zero. Solution\nIntroduce Laplace smoothing $$ \\phi_j = \\frac{1 + \\sum_{i=1}^{n} 1 { z^{(i)}=j }} {k+n} $$\nAdd $1$ to the numerator Add $k$ to the denominator $\\sum_{j=1}^{k} \\phi_j=1$ still holds $\\phi_j \\neq 0$ for all $j$ Under certain (strong) conditions\nLaplace smoothing gives the optimal estimator ","permalink":"https://andrelimzs.github.io/posts/machine-learning/generative-learning/","summary":"Generative Learning Generative learning is a different approach to learning as opposed to discriminative learning. It tries to model $p(x|y)$ and $p(y)$ instead of learning $p(y|x)$ directly.\n$p(x|y)$ : Distribution of the target\u0026rsquo;s features $p(y)$ : Class priors It uses Bayes rule to calculate posterior distribution $p(y|x)$ $$ p(y|x) = \\frac{p(x|y) p(y)}{p(x)} $$ And can be simplified when using for prediction because the denominator is constant and doesn\u0026rsquo;t matter $$ \\arg \\max_y P(y|x) = \\arg \\max_y p(x|y) p(y) $$","title":"Generative Learning"},{"content":"Generalised Linear Models (GLM) Generalised Linear Models (GLMs) are a family of models which include many common distributions such as Gaussian, Bernoulli and Multinomial.\nThe Exponential Family The exponential family serves as a starting point for GLMs.\nThe exponential family is defined as $$ p(y; \\eta) = b(y) \\ \\exp{\\left( \\eta^T\\ T(y) - a(\\eta) \\right)} $$\nNatural (Canonical) parameter : $\\eta$ Sufficient statistic : $T(y)$ Log Partition function : $a(\\eta)$ $T(y)$ is often chosen to be $T(y) = y$\n$e^{-a(\\eta)}$ serves as a normalisation constant, and ensures the distribution sums to $1$\nFamily / Set Of Distributions A family or set of distributions is defined by $(T, a, b)$ and parameterized by $\\eta$\nBernoulli as Exponential Family The Bernoulli distribution is in the exponential family. It is a distribution over $y \\in {0,1}$, with mean $\\phi$. Varying $\\phi$ gives different Bernoulli distributions.\nThe Bernoulli distribution can be shown to be an exponential family distribution by manipulating the equation into the form $p = b \\exp \\left( \\eta T - a \\right)$ $$ \\begin{aligned} p(y; \\phi) \u0026amp;= \\phi^y (1 - \\phi)^{1-y} \\cr \u0026amp;= \\exp \\left( y\\log\\phi + (1-y) \\log(1-\\phi) \\right) \\cr \u0026amp;= \\underbrace{1 \\vphantom{\\frac{}{}} }_b \\exp {\\large(} \\underbrace{\\log \\frac{\\phi}{1-\\phi} }_\\eta \\ \\underbrace{y \\vphantom{\\frac{}{}} }_T + \\underbrace{ \\log(1-\\phi) \\vphantom{\\frac{}{}} }_a {\\large)} \\end{aligned} $$\nGaussian as Exponential Family Similarly for the Gaussian distribution $$ \\begin{aligned} p(y; \\mu) \u0026amp;= \\frac{1}{\\sqrt{2\\pi}} \\left(-\\frac{1}{2} (y-\\mu)^2 \\right) \\cr \u0026amp;= \\underbrace{\\frac{1}{\\sqrt{2\\pi}} \\exp (-\\frac{1}{2} y^2)}_b \\cdot \\exp ( \\underbrace{\\mu \\vphantom{\\frac{}{}} }_n \\underbrace{y \\vphantom{\\frac{}{}} }_T - \\underbrace{\\frac{1}{2} \\mu^2}_a ) \\end{aligned} $$\nOther Distributions in the Exponential Family There are many other common distributions such as:\nMultinomial, Poisson Gamma, Exponential Beta, Dirichlet Constructing GLMs Consider a classification/regression problem where we want to predict $y$ as a function of $x$\nIf we make three assumptions\n1 . Given $x$ and $\\theta$, $y$ is some exponential family distribution with parameter $\\eta$ $$ y\\ |\\ x;\\theta \\sim \\text{ExpFamily}(\\eta) $$ 2 . Given $x$, predict $\\mathbb{E}[T(y)]$ $$ \\text{Want } h(x) = \\mathbb{E} [ T(y) \\ | \\ x] $$ 3 . Natural parameter $\\eta$ and inputs $x$ are linearly related $$ \\eta = \\theta^T x $$\nMany different/common types of distributions can be modelled as GLMs, and GLMs possess desirable properties such as ease of learning.\nCanonical Response Function : Distribution\u0026rsquo;s mean as function of the natural parameter $$ g(\\eta) = \\mathbb{E}[T(y); \\eta] $$ Canonical Link Function : Inverse of the response function $$ g^{-1} $$\nOrdinary Least Squares Ordinary least squares can be derived as a GLM with the following properties:\nTarget variable $y$ is Continuous $y$ given $x$ is modelled as Gaussian $\\mathcal{N}(\\mu, \\sigma^2)$ Formulate the hypothesis function from assumption 2 (Predict $\\mathbb{E}[T(y)]$ given $x$) $$ h_\\theta(x) = \\mathbb{E}[ y\\ |\\ x;\\theta] $$ Because it is Gaussian $$ h_\\theta(x) = \\mu $$ From the formulation of the Gaussian as an exponential family distribution, $\\mu = \\eta$ $$ \\begin{aligned} h_\\theta(x) \u0026amp;= \\mu \\cr \u0026amp;= \\eta \\end{aligned} $$ Finally from assumption 3 ($\\eta = \\theta^T x$) $$ \\begin{aligned} h_\\theta(x) \u0026amp;= \\eta \\cr \u0026amp;= \\theta^T x \\end{aligned} $$\nThe canonical response function of the Gaussian distribution is the identity function.\nLogistic Regression Logistic regression can also be formulated as a GLM.\n$y \\in {0, 1 }$\nChoose Bernoulli family to model $y\\ |\\ x$\nThe hypothesis function is $$ \\begin{aligned} h_\\theta(x) \u0026amp;= \\mathbb{E}[y\\ |\\ x;\\theta] \\cr \u0026amp;= \\phi \\vphantom{\\frac{1}{1}} \\cr \u0026amp;= \\frac{1}{(1 + e^{-\\eta})} \\cr \u0026amp;= \\frac{1}{(1+e^{-\\theta^Tx})} \\end{aligned} $$\nThe canonical response function of the Bernoulli distribution is the logistic function.\nSoftmax Regression Classification problem with multiple classes/categories Model as a multinomial distribution Express multinomial as an exponential family distribution\nParameterise Multinomial To parameterise over $k$ outcomes, we need to use $k-1$ parameters to prevent redundancy\n(The sum of all outcomes must be $1$)\nDefine $T(y) \\in \\mathbb{R}^{k-1}$ as $$ \\begin{aligned} T(1) \u0026amp;= { 1, 0, 0, \\dots, 0 } \\cr T(2) \u0026amp;= { 0, 1, 0, \\dots, 0 } \\cr T(k-1) \u0026amp;= { 0, 0, 0, \\dots, 1 } \\cr T(k) \u0026amp;= { 0, 0, 0, \\dots, 0 } \\end{aligned} $$ Define an indicator function: $$ \\left\\{ \\begin{aligned} \u0026amp;1 \u0026amp;\u0026amp;\\text{if True} \\cr \u0026amp;0 \u0026amp;\u0026amp;\\text{otherwise} \\end{aligned} \\right. $$\n$$ (T(y))_i = 1 \\{ y=i \\} $$\nWhich gives $$ \\begin{aligned} \\mathbb{E} [ (T(y))_i ] \u0026amp;= P(y=i) \\cr \u0026amp;= \\quad \\phi_i \\end{aligned} $$ Therefore the probability $$ \\begin{aligned} p(y; \\phi) \u0026amp;= \\phi_1^{1\\{y=1\\}} \\phi_2^{1\\{y=2\\}} \\dots \\phi_{k-1}^{1\\{y=k-1\\}} \\phi_k^{1-\\sum_{i=1}^{k-1}1\\{y=i\\}} \\cr \u0026amp;= \\phi_1^{(T(y))_1} \\phi_2^{(T(y))_2} \\dots \\phi_k^{1-\\sum_{i=1}^{k-1}1(T(y))_i} \\cr \u0026amp;= \\exp\\left( (T(y))_i \\log(\\phi_1) + \\dots + ((1 - \\Sigma_{i=1}^{k-1}(T(y))_i)) \\log(\\phi_k) \\right) \\end{aligned} $$\nFits the exponential family form $$ p(y;\\theta) = \\underbrace{1 \\vphantom{\\frac{1}{1}} }_b \\exp {\\LARGE(} \\underbrace{ (T(y))_i \\log(\\frac{\\phi_1}{\\phi_k}) + \\dots + (T(y))_{k-1} \\log(\\frac{\\phi_{k-1}}{\\phi_k}) }_\\eta + \\underbrace{ \\log(\\phi_k) \\vphantom{\\frac{1}{1}} }_a {\\LARGE)} $$\nWith:\nLink Function : $\\eta_i = \\log \\frac{\\phi_i}{\\phi_k}$\nResponse Function : $\\phi_i = \\frac{e^{n_i}}{\\sum_{j=1}^{k} e^{n_j}}$\nThis mapping from $\\eta \\rightarrow \\phi$ is also called the softmax function\nSoftmax regression is a generalisation of logistic regression\nIt outputs estimated probabilities for every value of $i = 1, \\dots, k$\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/generalised-linear-models/","summary":"Generalised Linear Models (GLM) Generalised Linear Models (GLMs) are a family of models which include many common distributions such as Gaussian, Bernoulli and Multinomial.\nThe Exponential Family The exponential family serves as a starting point for GLMs.\nThe exponential family is defined as $$ p(y; \\eta) = b(y) \\ \\exp{\\left( \\eta^T\\ T(y) - a(\\eta) \\right)} $$\nNatural (Canonical) parameter : $\\eta$ Sufficient statistic : $T(y)$ Log Partition function : $a(\\eta)$ $T(y)$ is often chosen to be $T(y) = y$","title":"Generalised Linear Models (GLM)"},{"content":"Classification Classification is similar to regression, except output $y$ only takes on a small number of discrete values, or classes.\nLogistic Regression Ignoring the fact that $y$ is discrete will often result in very poor performance. $h_\\theta(x)$ should also be constrained to $y \\in { 0, 1 }$\nOne approach is to modify the hypothesis function to use the logistic / sigmoid function $$ h_\\theta(x) = g(\\theta^Tx) = \\frac{1}{1 + e^{-\\theta^Tx}} $$ where $g(z) = \\frac{1}{1 + e^{-z}}$ is the logistic/sigmoid function\nProperties of Sigmoid\n$g(z)$ tends to: $$ \\left\\{ \\begin{aligned} 0 \u0026amp;\u0026amp; z \\rightarrow -\\infty \\cr 1 \u0026amp;\u0026amp; z \\rightarrow \\infty \\end{aligned} \\right. $$ The derivative is $$ g\u0026rsquo; = g(1 - g) $$\nProbabilistic Interpretation Similar to least-square regression, the classification model can be derived as a maximum likelihood estimator of $\\theta$\nAssume $$ \\begin{aligned} P(y=1 | x; \\theta) \u0026amp;= h_\\theta(x) \\cr P(y=0 | x; \\theta) \u0026amp;= 1 - h_\\theta(x) \\end{aligned} $$ or equivalently $$ P(y | x; \\theta) = \\left( h_\\theta(x) \\right)^y \\ \\left( 1-h_\\theta(x) \\right)^{1-y} $$ For $n$ independent training examples the likelihood function (distribution of $y$ given $x$ and parameterised by $\\theta$) is $$ \\begin{aligned} L(\\theta) \u0026amp;= p(y | X;\\theta) \\cr \u0026amp;= \\prod_{i=1}^{n} p(y^{(i)} | x^{(i)}; \\theta) \\cr \u0026amp;= \\prod_{i=1}^{n} \\left( h_\\theta(x^{(i)}) \\right)^{y^{(i)}} \\left(1 - h_\\theta(x^{(i)}) \\right)^{1-y^{(i)}} \\end{aligned} $$ Which can be transformed into log likelihood $$ l(\\theta) = \\sum_{i=1}^{n} y^{(i)} \\log h(x^{(i)}) + (1 - y^{(i)}) \\log(1 - h(x^{(i)})) $$ And maximised using gradient ascent\nGradient Ascent rule The update rule is $$ \\frac{\\partial}{\\partial \\theta_j} l(\\theta) = (y - h_\\theta(x)) \\ x_j $$\nWhich gives the stochastic gradient ascent rule $$ \\theta_j := \\theta_j + \\alpha \\left( y^{(i)} - h_\\theta(x^{(i)}) \\right)\\ x_j^{(i)} $$\nOther Optimizers - Newton\u0026rsquo;s Method An alternative method to gradient descent is newton\u0026rsquo;s method. It uses the fact that the maxima of a function is the point where the first derivative is zero $$ \\theta := \\theta - \\frac{l\u0026rsquo;(\\theta)}{l\u0026rsquo;\u0026rsquo;(\\theta)} $$\nVector-Valued Generalisation The generalisation is called the Newton-Raphson method $$ \\theta = \\theta - H^{-1} \\nabla_\\theta l(\\theta) $$ Where $\\nabla_\\theta l(\\theta)$ is the Jacobian and $H$ is the Hessian $$ H_{ij} = \\frac{\\partial^2 l(\\theta)}{\\partial \\theta_i \\ \\partial \\theta_j} $$\nComparison with Gradient Descent Advantages\nFaster convergence Requires significantly fewer iterations Disadvantages\nEach iteration can be more expensive Requires inverting the Hessian ","permalink":"https://andrelimzs.github.io/posts/machine-learning/supervised-learning-classification/","summary":"Classification Classification is similar to regression, except output $y$ only takes on a small number of discrete values, or classes.\nLogistic Regression Ignoring the fact that $y$ is discrete will often result in very poor performance. $h_\\theta(x)$ should also be constrained to $y \\in { 0, 1 }$\nOne approach is to modify the hypothesis function to use the logistic / sigmoid function $$ h_\\theta(x) = g(\\theta^Tx) = \\frac{1}{1 + e^{-\\theta^Tx}} $$ where $g(z) = \\frac{1}{1 + e^{-z}}$ is the logistic/sigmoid function","title":"Supervised Learning - Classification"},{"content":"Supervised Learning Supervised learning is the task of learning a function mapping from input to output $$ y = f(x) $$ The relationship can be linear/nonlinear or convex/nonconvex. The approach learns from labeled data.\nTerminology Input (Features) : $x^{(i)}$\nOutput (Target) : $y^{(i)}$\nTraining example : $(x^{(i)}, y^{(i)})$\nHypothesis : $h(x)$\nParameters / Weights : $\\theta$\nTypes Regression : Continuous values\nClassification : Discrete values\nLinear Regression Objective\nLearn parameters $\\theta$ for a given hypthesis function $h$ to best predict output $y$ from input $x$\nHypothesis Function\nCandidate model that we think will best fit the data\n[Example] A linear model can be represented as $$ \\begin{aligned} h(x) \u0026amp;= \\sum_{i=0}^d \\theta_i x_i \\cr \u0026amp;= \\theta^T x \\end{aligned} $$\nCost / Loss Function\nMeasure of the accuracy or error of the model. The cost is driven to zero to fit the model to the data.\n[Example] A common cost function is least-squares $$ J(\\theta) = \\frac{1}{2} \\sum_{i=1}^{n} ( h_\\theta (x^{(i)} - y^{(i)}))^2 $$ which results in ordinary least squares regression\nGradient Descent An optimisation algorithm commonly used to fit machine learning models. It is an iterative, first-order approach. It is also known as the Least mean square (LMS) update rule or Widrow-Hoff learning rule.\nThe update (for single training example) $$ \\theta_j := \\theta_j - \\alpha \\frac{\\partial}{\\partial \\theta_j} J(\\theta) $$\nmoves $\\theta$ in the direction of best improvement\nBatch Gradient Descent More generally, gradient descent can be applied to multiple training examples simultaneously by grouping updates into vector $x$ $$ \\theta := \\theta + \\alpha \\underbrace{\\sum_{i=1}^{n} \\left( y^{(i)}- h_\\theta(x^{(i)}) \\right) x^{(i)} }_{\\large = \\frac{\\partial J}{\\partial \\theta} \\quad} $$ Which is equivalent to gradient descent on the original cost function $J$.\nBatch gradient descent looks at every example on every iteration and can therefore be slow.\nStochastic Gradient Descent (SGD) An alternative to batch gradient descent is stochastic gradient descent (also known as incremental gradient descent).\nSGD updates the weights $\\theta$ on every iteration, making much quicker progress for large datasets.\nThe update rule is given by $$ \\theta := \\theta + \\alpha \\left( y^{(i)}- h_\\theta(x^{(i)}) \\right) x^{(i)} $$\nwhich is repeatedly run through a loop\nBatch vs SGD Batch SGD Slower convergence Faster convergence Converge to minimum Oscillate about minimum In most cases being close to the minimum is good enough, and therefore people choose SGD for the faster convergence\nThe Normal Equations (Explicit Minimisation) In the case of linear regression, $J$ can be minimised explicitly by taking the derivatives w.r.t $J$ and setting them to zero\nMatrix Derivatives : For function $f : \\mathbb{R}^{n \\times d} \\rightarrow \\mathbb{R}$ the Jacobian is $\\nabla_A f(A)$\nLeast-Squares via Moore-Penrose Psuedoinverse Set $\\nabla_\\theta J(\\theta) = 0$\n$$ \\underbrace{\\nabla_\\theta J(\\theta)}_{= 0} = X^TX \\theta - X^T y $$ Therefore $$ X^TX \\theta = X^T y $$ Which gives the closed-form solution $$ \\theta = (X^TX)^{-1} X^T y $$\nProbabilistic Interpretation Linear regression with least-squares cost can be derived from statistical methods to give a probabilistic interpretation.\nAssume target and inputs are related via $$ y^{(i)} = \\theta^T x^{(i)} + \\underbrace{ \\epsilon^{(i)} }_\\text{noise} $$ where $\\epsilon^{(i)}$ are unmodelled effects/noise\nAssume $\\epsilon^{(i)}$ are Gaussian IID $$ p(\\epsilon^{(i)}) = \\frac{1}{\\sqrt{2\\pi\\sigma}} \\exp \\left( - \\frac{(\\epsilon^{(i)})^2}{2\\sigma^2} \\right) $$ Which implies that $y$ given $x$ parameterised by $\\theta$ is gaussian $$ p(y^{(i)} | x^{(i)};\\theta) = \\frac{1}{\\sqrt{2\\pi\\sigma}} \\exp \\left( - \\frac{ (y^{(i)} - \\theta^T x^{(i)})^2 } { 2\\sigma^2 } \\right) $$ or $$ y^{(i)} | x^{(i)}; \\theta \\sim \\mathcal{N}(\\theta^T x ^{(i)}, \\sigma^2) $$\nLikelihood Function The likelihood function answers the question\nGiven $X$ and $\\theta$, what is the distribution of $y^{(i)}$s?\n$$ \\begin{aligned} L(\\theta) \u0026amp;= L(\\theta; X, \\vec{y}) \\cr \u0026amp;= p(\\vec{y}|X;\\theta) \\end{aligned} $$ Can be written as $$ \\begin{aligned} L(\\theta) \u0026amp;= \\prod_{i=1}^{n} p(y^{(i)} | x^{(i)}; \\theta) \\cr \u0026amp;= \\prod_{i=1}^{n} \\frac{1}{\\sqrt{2\\pi\\sigma}} \\exp \\left( - \\frac{ (y^{(i)} - \\theta^T x^{(i)})^2 } { 2\\sigma^2 } \\right) \\end{aligned} $$\nMaximum Likelihood Estimation (MLE) of $\\theta$ The Principle of maximum likelihood says that we should choose $\\theta$ to maximise $L(\\theta)$ (make the probability of $y$ given $x$ as high as possible)\nTo simplify the procedure, the we can maximise any strictly increasing function of $L(\\theta)$ such as log likelihood, $l(\\theta)$. $$ l(\\theta) = n \\log \\frac{1}{\\sqrt{2\\pi\\sigma}} - \\frac{1}{2\\sigma^2} \\sum_{i=1}^{n} ( y^{(i)} - \\theta^T x^{(i)} )^2 $$ Which is the same as minimizing the least-squares costs $$ J(\\theta) = \\frac{1}{2} \\sum_{i=1}^{n} ( y^{(i)} - \\theta^T x^{(i)} )^2 $$\nLocally Weighted Linear Regression (LWR) The choice of features is important because it can result in overfitting or underfitting.\nCS229 https://cs229.stanford.edu/\nBut one method to make the choice of features less critical is locally weighted linear regression (LWR), assuming there is sufficient training data.\nProcedure\nFit $\\theta$ to minimise $\\sum_i w^{(i)} ( y^{(i)} - \\theta^T x^{(i)} )^2$ Output $\\theta^T x$ [*] Only difference is the weight $w^{(i)}$ $$ w^{(i)} = \\exp \\left( -\\frac{(x^{(i)} - x)^2}{2\\tau^2} \\right) $$ Which biases around the query point\nBandwidth : $\\tau$, controls the falloff distance\nThis is an example of a non-parametric Algorithm\nRequires data even after fitting Non-parametric means hypothesis $h$ grows linearly with size of training set ","permalink":"https://andrelimzs.github.io/posts/machine-learning/supervised-learning-regression/","summary":"Supervised Learning Supervised learning is the task of learning a function mapping from input to output $$ y = f(x) $$ The relationship can be linear/nonlinear or convex/nonconvex. The approach learns from labeled data.\nTerminology Input (Features) : $x^{(i)}$\nOutput (Target) : $y^{(i)}$\nTraining example : $(x^{(i)}, y^{(i)})$\nHypothesis : $h(x)$\nParameters / Weights : $\\theta$\nTypes Regression : Continuous values\nClassification : Discrete values\nLinear Regression Objective\nLearn parameters $\\theta$ for a given hypthesis function $h$ to best predict output $y$ from input $x$","title":"Supervised Learning - Regression"},{"content":"Review of Probability Theory Overview Probability is an important aspect of machine learning, and forms a useful basis for concepts. It is the study of uncertainty\nElements of Probability Defining probability on a set requires the following:\nSample Space : $\\Omega$\nThe set of all possible outcomes of a random experiment Each outcome $\\omega \\in \\Omega$ is the complete description of the state at the end of the experiment Event Space (Set of events) : $\\mathcal{F}$\nA set whose elements $A \\in \\mathcal{F}$ (events) are subsets of $\\Omega$ Eg: $A$ is the set of possible outcomes for an experiment Probability Measure : $P : \\mathcal F \\rightarrow \\mathbb R$ which satisfies the Axioms of Probability\n$P(A) \\geq 0$ for all $A \\in \\mathcal F$ $P(\\Omega) = 1$ If $A_1, A_2, \\dots$ are disjoint events $P(\\cup_i A_i) = \\sum_i P(A_i)$ Conditional Probability For an event $B$ with non-zero probability\nThe conditional probability of $A$ given $B$ is $$ P(A|B) \\triangleq \\frac{P (A \\cap B)}{P(B)} $$\nIndependence If and only if\n$P(A \\cap B) = P(A) P(B)$ $P(A|B) = P(A)$ Random Variables A function $X : \\Omega \\rightarrow \\mathbb R$ that maps a sample space to a real-valued outcome\nCumulative Distribution Functions (CDF) Function $F_X : \\mathbb R \\rightarrow [0,1]$ that calculates the probability of an event $$ F_X(x) \\triangleq P(X \\leq x) $$ Properties\n$0 \\leq F_X(x) \\leq 1$ $\\underset{x \\rightarrow -\\infty}{\\lim} F_X(x) = 0$ $\\underset{x \\rightarrow \\infty}{\\lim} F_X(x) = 1$ $x \\leq y \\implies F_X(x) \\leq F_Y(y)$ Probability Mass Function (PMF) Function $p_X : \\Omega \\rightarrow \\mathbb R$ that directly specifies the probability of each value for a discrete RV $$ p_X(x) \\triangleq P(X=x) $$ Properties\n$0 \\leq p_X(x) \\leq 1$ $\\underset{x\\in Val(X)}{\\sum} p_X(x) = 1$ $\\underset{x\\in A}{\\sum} p_X(x) = P(X\\in A)$ Probability Density Functions (PDF) Derivative of the CDF $$ f_X(x) \\triangleq \\frac{dF_X(x)}{dx} $$ If the CDF is differentiable everywhere\nProperties\n$f_X(x) \\geq 0$ $\\int_{-\\infty}^{\\infty} f_X(x) = 1$ $\\int_{x\\in A} f_X(x) dx = P(X \\in A)$ Expectation The \u0026ldquo;weighted average\u0026rdquo; of the values that a distribution can take\nDiscrete RV $$ E[g(X)] \\triangleq \\sum_{x\\in Val(A)} g(x) p_X(x) $$ Continuous RV $$ E[g(X)] \\triangleq \\int_{-\\infty}^{\\infty} g(x) f_X(x)\\ dx $$ Properties\n$E[a] = a$ for any constant $a$ $E[a f(X)] = aE[f(X)]$ for any constant $a$ (Linearity of Expectation) $E[f(X) + g(X)] = E[f(X)] + E[g(X)]$ For discrete RV $E[1{ X=k }] = P(X = k)$ Variance Measure of how concentrated a distribution is around it\u0026rsquo;s mean $$ Var[X] \\triangleq E[(X - E(X))^2] $$ or equivalently $$ Var[x] = E[X^2] - E[X]^2 $$ Properties\n$Var[a] = 0$ for any constant $a$ $Var[a f(X)] = a^2 Var[f(X)]$ for any constant $a$ Common Random Variables Discrete Bernoulli\nEvent happens with probability $p$ $$ p(x) = \\left\\{ \\begin{aligned} \u0026amp;p \u0026amp;\u0026amp; \\text{if } p = 1 \\cr \u0026amp;1-p \u0026amp;\u0026amp; \\text{if } p = 0 \\end{aligned} \\right. $$ Binomial\nNumber of outcomes in $n$ independent events $$ p(x) = \\begin{pmatrix} n \\cr x \\end{pmatrix} p^x (1-p^{n-x}) $$ Geometric\nNumber of independent events until outcome is true $$ p(x) = p (1-p)^{x-1} $$ Poisson\nProbability distribution over nonnegative integers $$ p(x) = e^{-\\lambda} \\frac{\\lambda^x}{x!} $$\nContinuous Uniform\nEqual probability density to every value between $a$ and $b$ $$ f(x) = \\left\\{ \\begin{aligned} \u0026amp;\\frac{1}{b-a} \u0026amp;\u0026amp; \\text{if } a\\leq x \\leq b \\cr \u0026amp;0 \u0026amp;\u0026amp; \\text{otherwise} \\end{aligned} \\right. $$\nExponential\nDecaying probability density over nonnegative reals $$ f(x) = \\left\\{ \\begin{aligned} \u0026amp;\\lambda e^{-\\lambda x} \u0026amp;\u0026amp; \\text{if } x\\geq 0 \\cr \u0026amp;0 \u0026amp;\u0026amp; \\text{otherwise} \\end{aligned} \\right. $$ Normal\nGaussian distribution $$ f(x) = \\frac{1}{\\sqrt{2\\pi}\\sigma} \\exp \\left( -\\frac{1}{2\\sigma^2}(x-\\mu)^2 \\right) $$\nJoint and Marginal Distributions CDF Of $X$ and $Y$ $$ F_{XY}(x,y) = P(X \\leq x, Y \\leq y) $$ The joint CDF is related to the marginal CDF $F_X(x) / F_Y(y)$ via $$ \\begin{aligned} F_X(x) \u0026amp;= \\lim_{y \\rightarrow \\infty} F_{XY}(x,y) dy \\cr F_Y(y) \u0026amp;= \\lim_{x \\rightarrow \\infty} F_{XY}(x,y) dx \\end{aligned} $$ Properties\n$0 \\leq F_{XY}(x,y) \\leq 1$ $\\lim_{x,y \\rightarrow \\infty} F_{XY}(x,y) = 1$ $\\lim_{x,y \\rightarrow -\\infty} F_{XY}(x,y) = 0$ $F_X(x) = \\lim_{y \\rightarrow \\infty} F_{XY}(x,y)$ PMF $$ p_{XY}(x,y) = P(X=x, Y=y) $$\nAnd is related to the marginal PMF via $$ \\begin{aligned} p_X(x) \u0026amp;= \\sum_y p_XY(x,y) \\cr p_Y(y) \u0026amp;= \\sum_x p_XY(x,y) \\end{aligned} $$\nPDF $$ f_{XY}(x,y) = \\frac {\\partial^2 F_{XY}(x,y)} {\\partial x \\partial y} $$\nWhere $$ \\begin{aligned} \\iint_{x\\in A} f_XY(x,y)\\ dx\\ dy = P((X,Y) \\in A) \\cr \\end{aligned} $$ And $$ \\begin{aligned} f_X(x) = \\int_{-\\infty}^{\\infty} f_{XY}(x,y) dy \\cr f_Y(y) = \\int_{-\\infty}^{\\infty} f_{XY}(x,y) dx \\end{aligned} $$\nConditional Distribution What is the probability distribution over $Y$ given that we know $X = x$?\nDiscrete $$ p_{Y|X} (y|x) = \\frac{p_{XY}(x,y)}{p_X(x)} $$ Continuous $$ f_{Y|X} (y|x) = \\frac{f_{XY}(x,y)}{f_X(x)} $$\nBayes\u0026rsquo;s Rule Used to derive conditional probability of one variable given another $$ P(A|B)\\ P(B) = P(B|A)\\ P(A) $$ And therefore $$ P(A|B) = \\frac{ P(B|A)\\ P(A) }{ P(B) } $$\nIndependence Two RVs $X$ and $Y$ are independent if $$ P(X,Y) = P(X)\\ P(Y) $$ for all values of $x$ and $y$\nExpectation and Covariance Let $X$ and $Y$ be RVs and $g : \\mathbb{R}^2 \\rightarrow \\mathbb R$\nExpectation Discrete $$ E{g(X,Y)} \\triangleq \\sum_{x \\in Val(X)} \\sum_{y \\in Val(Y)} g(x,y) p_{XY}(x,y) $$ Continuous $$ E[g(X,Y)] = \\int_{-\\infty}^{\\infty} \\int_{-\\infty}^{\\infty} g(x,y) f_{XY}(x,y)\\ dx\\ dy $$\nConvariance $$ \\begin{aligned} Cov[X,Y] \u0026amp;\\triangleq E[(X-E[X])(Y-E(Y))] \\cr \u0026amp;= E[XY] - E[X] E[Y] \\end{aligned} $$\nUncorrelated : $Cov[X,Y] = 0$\nProperties\n(Linearity of Expectation) $Var[X+Y] = Var[ X ] + Var[Y] + 2 Cov[X,Y]$ If $X$ and $Y$ independent $\\implies E[f(X)g(Y)] = E[f(X)]\\ E[g(Y)]$ Multiple RV The notation for two RVs can be generalised to $n$ RVs\nProperties Chain Rule $$ f(x_1, x_2, \\dots, x_n) = f(x_1) f(x_2 | x_1) \\prod_{i=3}^{n} f(x_i | x_1, \\dots, x_{i-1}) $$\nIndependence Events $A_1, \\dots, A_k$ are mutually independent if $$ P(\\underset{i \\in S}{\\cap} A_i) = \\prod_{i\\in S} P(A_i) $$ Random variables $X_1, \\dots, X_n$ are independent if $$ f(x_1, \\dots, x_n) = f(x_1) f(x_2) \\dots f(x_n) $$\nRandom Vectors A vector of random variables is a random vector\nIt is a mapping from $\\Omega \\rightarrow \\mathbb R^n$\nIt is an alternative notation for multiple RVs $$ X = \\begin{bmatrix} X_1 \\cr \\vdots \\cr X_n \\end{bmatrix} $$\nExpectation $$ E[g(X)] = \\begin{bmatrix} E[g_1(X)] \\cr \\vdots \\cr E[g_m(X)] \\end{bmatrix} $$\nCovariance $$ \\Sigma = \\begin{bmatrix} E[X_1^2] \u0026amp; \\dots \u0026amp; E[X_1 X_n] \\cr \\vdots \u0026amp; \\ddots \u0026amp; \\vdots \\cr E[X_n X_1] \u0026amp; \\dots \u0026amp; E[X_n^2] \\end{bmatrix} $$\nor $$ \\begin{aligned} \\Sigma \u0026amp;= E[XX^T] - E[X]\\ E[X]^T \\cr \u0026amp;= E[(X-E[X])(X-E[X])^T] \\end{aligned} $$\nProperties\nPositive semidefinite : $\\Sigma \\succeq 0$ Symmetric : $\\Sigma = \\Sigma^T$ Multivariate Gaussian Distribution Particularly important example\nHas a\nMean : $\\mu \\in \\mathbb R^n$\nCovariance : $\\Sigma \\in \\mathbb S_{++}^n$ (symmetric positive definite matrices) $$ f_{X_1, \\dots, X_n} (x_1, \\dots, x_n; \\mu, \\Sigma) = \\frac{1}{(2\\pi)^{\\pi/2} |\\Sigma|^{1/2} } \\exp \\left( -\\frac{1}{2} (x-\\mu)^T \\Sigma^{-1} (x-\\mu) \\right) $$ Written as $X \\sim \\mathcal N(\\mu, \\Sigma)$\nUsefulness\nComes up extremely often due to the Central Limit Theorem (CLT) - A large number of independent RVs will tend toward a Gaussian distribution.\nUseful because it has simple closed-form solutions.\n","permalink":"https://andrelimzs.github.io/posts/machine-learning/cs229-0-probability-theory/","summary":"Review of Probability Theory Overview Probability is an important aspect of machine learning, and forms a useful basis for concepts. It is the study of uncertainty\nElements of Probability Defining probability on a set requires the following:\nSample Space : $\\Omega$\nThe set of all possible outcomes of a random experiment Each outcome $\\omega \\in \\Omega$ is the complete description of the state at the end of the experiment Event Space (Set of events) : $\\mathcal{F}$","title":"Probability Theory"},{"content":"I\u0026rsquo;ve finished up the machine, and sold it. On to other projects.\n","permalink":"https://andrelimzs.github.io/cnc-machine-conclusion/","summary":"I\u0026rsquo;ve finished up the machine, and sold it. On to other projects.","title":"CNC Machine Conclusion"},{"content":"1) 2F, 4mm DOC, 1.6mm WOC, 0.02 FPT, 150 m/min, 90N Too much DOC/WOC and not enough feed-per-tooth? Significant chatter\n2) 1F, 2mm DOC, 0.6mm WOC, 0.08 FPT, 150m/min, 45N Dropping DOC \u0026amp; WOC and increasing FPT seems to reduce chatter\n3) 1F, 1.2mm DOC, 0.4mm WOC, 0.16 FPT, 75m/min, 54N Even less chatter\n4) Non-adaptive climb milling 1F, 0.8mm DOC, 0.4mm WOC, 0.16 FPT, 150m/min, 72N Looks good\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-12-14-first-cuts/","summary":"1) 2F, 4mm DOC, 1.6mm WOC, 0.02 FPT, 150 m/min, 90N Too much DOC/WOC and not enough feed-per-tooth? Significant chatter\n2) 1F, 2mm DOC, 0.6mm WOC, 0.08 FPT, 150m/min, 45N Dropping DOC \u0026amp; WOC and increasing FPT seems to reduce chatter\n3) 1F, 1.2mm DOC, 0.4mm WOC, 0.16 FPT, 75m/min, 54N Even less chatter\n4) Non-adaptive climb milling 1F, 0.8mm DOC, 0.4mm WOC, 0.16 FPT, 150m/min, 72N Looks good","title":"First Cuts"},{"content":"Frame This was the basic frame, made up of only extrusions and joints. I added the linear modules and custom plates, spindle and fixture plate Finally the enclosure, motors and limit switches Electronics I can\u0026rsquo;t be sure of the quality of these power supplies, so I\u0026rsquo;m using one for the steppers and another for the spindle. I hope this prevents noise from the spindle affecting the steppers.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-08-09-assembling-the-machine/","summary":"Frame This was the basic frame, made up of only extrusions and joints. I added the linear modules and custom plates, spindle and fixture plate Finally the enclosure, motors and limit switches Electronics I can\u0026rsquo;t be sure of the quality of these power supplies, so I\u0026rsquo;m using one for the steppers and another for the spindle. I hope this prevents noise from the spindle affecting the steppers.","title":"Assembling the machine"},{"content":"G2Core is a motion control system, similar to grbl, mach3/4, linuxCNC, etc. It runs on a 32-bit ARM Cortex M3 Arduino Due, which means plenty of power for additional features.\nDue Pinout\nConfiguration in G2core Term Meaning machine BOARD + SETTINGS_FILE CONFIG Specifies default BOARD/SETTINGS_FILE for a machine BOARD Specify board type and revision BASE_BOARD Specify underlying hardware platform SETTINGS_FILE Specify default settings Enable Motors and Axes Create a new SETTINGS_FILE, settings_defaultdue.h\nEnable motors M1_POWER_MODE: MOTOR_ALWAYS_POWERED\nSet motor power level M1_POWER_LEVEL: 1.0\nEnable axes X_AXIS_MODE: AXIS_STANDARD\nBuild and Flash Add a new config in boards.mk which will use the new settings_defaultdue.h ifeq (\u0026#34;\u0026amp;#36;(CONFIG)\u0026#34;,\u0026#34;DefualtDue\u0026#34;) ifeq (\u0026#34;\u0026amp;#36;(BOARD)\u0026#34;,\u0026#34;NONE\u0026#34;) BOARD=g2v9k endif SETTINGS_FILE=\u0026#34;settings_defaultdue.h\u0026#34; endif Clean then Build Solution\nFlash the board using arduino-flash-tools, which in my case is\nmode COM3 BAUD=1200 bossac.exe --port=COM3 -U false -e -w -v -b C:\\Github\\g2\\g2core\\bin\\DefualtDue-gShield\\g2core.bin -R Connect Switch the USB cable to the other port on the board, and connect to CNCjs/UGS/Chilipeppr ","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-19-g2core-motion-controller/","summary":"G2Core is a motion control system, similar to grbl, mach3/4, linuxCNC, etc. It runs on a 32-bit ARM Cortex M3 Arduino Due, which means plenty of power for additional features.\nDue Pinout\nConfiguration in G2core Term Meaning machine BOARD + SETTINGS_FILE CONFIG Specifies default BOARD/SETTINGS_FILE for a machine BOARD Specify board type and revision BASE_BOARD Specify underlying hardware platform SETTINGS_FILE Specify default settings Enable Motors and Axes Create a new SETTINGS_FILE, settings_defaultdue.","title":"G2Core Motion Controller"},{"content":"I stumbled on an excellent resource, Principles of Rapid Machine Design Bamberg, E. (2000). Principles of rapid machine design. It covers design and manufacturing principles, calculations and simulations, and experimental data.\nDesign Principles Stiffness Budget \u0026ldquo;One of the most important criteria is the effective stiffness of the tool/work piece interface, measured in force per unit deflection $N/\\mu m$.\u0026rdquo; Bamberg states that values of 10 to 25 are desirable for machining.\nThe idea behind the stiffness budget is to assume each sub-assembly acts as a spring (of finite stiffness) and calculate the total stiffness from\n$$ k_{tot} = \\left( \\sum_{i=1}^{n} \\frac{1}{k_i} \\right)^{-1} $$\nThe total stiffness of springs in series is given by the inverse of the sum of the reciprocals. Recursively apply this concept on each component in the sub-assembly until every sub-component is accounted for.\nI graphed $k_{tot}$ as a function of $k_1$ for varying $k_2/k_3$. Because it’s the inverse sum of reciprocals\nOne weak component can quickly bring stiffness to zero One exceptionally strong component has limited effect Which means that the optimal design will be something like equal stiffness for every component?\nFEA Modeling Techniques Bearings \u0026ldquo;A bearing block modeled from (\u0026hellip;) steel or even aluminum will appear much stiffer than it actually is\u0026rdquo;. Bamberg offers two solutions:\nModify the geometry to get accurate deflection Modify the material properties He did this by calculating an equivalent Young\u0026rsquo;s modulus.\nManufacturing Principles Aluminum profiles suffer from \u0026ldquo;excessive thermal expansion\u0026rdquo; and \u0026ldquo;limited strength of the joints\u0026rdquo; which limit it to \u0026ldquo;low force operations such as light machining\u0026rdquo;.\nBamberg raised a point that the structure a bearing rail is attached to has to be stiffer than the hardened steel rail. Otherwise the rail \u0026ldquo;would warp the structure rather than [be] straightened by it\u0026rdquo;.\nThese are potentially problematic. Putting aside thermal expansion and rail warping, I might be able to get stronger joints by making them bigger and accommodate multiple bolts.\nDamping Real machines are too complicated to be modeled as simple spring-mass systems. A common approach is loss factor $\\eta$ , which is the ratio of average energy dissipated (per radian) to peak energy (per cycle).\nMaterial Damping Principles of Rapid Machine Design\nAluminum has something like half the loss factor of mild steel, which might be a problem. But bolted joints provide damping because friction along the joint dissipates energy. \u0026ldquo;This type of damping is cumulative and therefore increases with the number of joints\u0026rdquo;.\nPrinciples of Rapid Machine Design\nIt is interesting that the joints and bearings provide 80 – 90% of the total damping. This means that using aluminum instead of steel is only a difference of 5 - 10%?\nSomething to investigate is the damping of a t-slot nut (normal and long variants) vs tapping a hole directly in the profile.\nConclusion Avoid any single point of low stiffness because of $k_{tot} = \\left( \\sum_{i=1}^{n} \\frac{1}{k_i} \\right)^{-1}$ Joints are the weakest link in aluminum profile structures Aluminum has half the material damping of steel but 80 - 90% of total machine damping comes from bolts and bearings ","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-18-principles-of-rapid-machine-design-bamberg/","summary":"I stumbled on an excellent resource, Principles of Rapid Machine Design Bamberg, E. (2000). Principles of rapid machine design. It covers design and manufacturing principles, calculations and simulations, and experimental data.\nDesign Principles Stiffness Budget \u0026ldquo;One of the most important criteria is the effective stiffness of the tool/work piece interface, measured in force per unit deflection $N/\\mu m$.\u0026rdquo; Bamberg states that values of 10 to 25 are desirable for machining.","title":"Principles of Rapid Machine Design – Bamberg"},{"content":" The parts are going to take a while to arrive, so in the meantime I\u0026rsquo;m refining my design again. I went from a XY-Z configuration to Y-XZ. This frees up space in both the x and z axes and should make it easier to put additional reinforcement as well as a future 4th axis.\nThe y-axis table also extends out of the frame, which will make it easier to load/unload the machine.\nReinforcing the X-axis For the x-axis, a force on the spindle (red arrow)\nResults in a moment about the z-axis (blue arrows)\nWhich depends on the polar moment of area on the XY-plane (green shaded)\nReinforcing the Y-axis A force on the y-axis (red arrow)\nResults in a moment (blue arrow)\nWhich depends on the polar moment of area in the YZ-plane (green shaded)\nFront Column Both x- and y-axis benefit from adding a second column in the front.\nIt more than triples the polar moment of area, if some assumptions are met (joints, rails \u0026amp; ballscrews are significantly stronger than the extrusion etc).\nFEA Simulation I started using the \u0026ldquo;trend tracker\u0026rdquo; feature in solidworks. It\u0026rsquo;s convenient and a real time saver.\nAdding the front column led to the most significant improvement, from 150 μm to 65 μm.\nAdding the 5mm triangle/L-shaped plates dropped it further to 20 μm. The difference between aluminum (6061) and stainless steel (304) was negligible.\nFor the second study I changed the fixtures because the model was becoming numerically unstable. The table is still fixed and the force is still a remote load acting on the spindle mount but I added an elastic support on the base to simulate the effects of gravity/the ground. Changing the fixture immediately dropped deflection from 20 μm to 8 μm. It makes sense that the weight of the machine will help anchor it to the ground.\nAdding a second horizontal beam between front and back columns halved the deflection, to 4 μm.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-15-cnc-mill-v3/","summary":"The parts are going to take a while to arrive, so in the meantime I\u0026rsquo;m refining my design again. I went from a XY-Z configuration to Y-XZ. This frees up space in both the x and z axes and should make it easier to put additional reinforcement as well as a future 4th axis.\nThe y-axis table also extends out of the frame, which will make it easier to load/unload the machine.","title":"CNC Mill V3"},{"content":" When we last left off we managed to get 20 μm, I\u0026rsquo;m going to try to improve on it.\nSpindle Brace Adding\nStainless steel sheet metal (5 mm) brace between the spindle and rear column Aluminum sheet (4 mm) at the bottom reduces deflection (at 110 N) from 20 μm to 16 μm.\nColumn Brace Adding a 60 x 200 x 5 mm stainless steel plate reduces deflection from 16 μm to 14.3 μm.\nCustom Column Plate Removing the spindle brace, the deflection goes back up to 17.4 μm. Each triangle plate from Misumi is $$$40, and the custom plate replaces three of them. If the custom plates cost less than $$$120, it\u0026rsquo;ll be a good deal.\nCombine Column and Triangle Plate Goes down slightly to 17.1 μm, and it\u0026rsquo;ll save $$$80.\nEnclosure The enclosure will\nKeep the noise down Contain the mess People have managed to get their Shapeoko and Nomad machines under 56dB (Carbide3d Forum).\nI have four options for the enclosure panels\nAluminum (3 mm) – $$$106/pc Polycarbonate (5 mm) – $$$81/pc Powder Coat Steel (2 mm, 8 hole) – $$$72/pc Stainless Steel (2 mm) – $$$119 3 and 5 mm panels can be mounted inside the extrusions, everything else has to be bolted on. I\u0026rsquo;ll need to investigate how that affects noise isolation.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-10-improving-rigidity/","summary":"When we last left off we managed to get 20 μm, I\u0026rsquo;m going to try to improve on it.\nSpindle Brace Adding\nStainless steel sheet metal (5 mm) brace between the spindle and rear column Aluminum sheet (4 mm) at the bottom reduces deflection (at 110 N) from 20 μm to 16 μm.\nColumn Brace Adding a 60 x 200 x 5 mm stainless steel plate reduces deflection from 16 μm to 14.","title":"Improving Rigidity"},{"content":" Design Goal (Quantified) After a few more weeks of research and reading up I have decided to use deflection at the spindle as my primary design goal. I plan to use depth of cut (DOC) + width of cut (WOC) to estimate the acceptable deflection.\nDepth of Cut DOC in aluminum ranges from 0.254mm/0.76mm/1.6mm (Shapeoko Wiki) to 2.54mm (Vince Fab) and 6.35mm (Shapeoko Forum).\nRecommendations for the Tormach PCNC1100 is also 6.35mm (CNC Zone).\nWOC is recommended to be between 0.1 - 0.5 of the endmill diameter.\nCutting Force There is a chart on the Shapeoko forum that says that a 500W spindle will see a force of 4.5N (Shapeoko Forum). This seems extremely conservative.\nUsing the Kennametal milling calculator, and 0.02mm feed per tooth (FPT or chipload), and 220/150 m/min cutting speed (SFM)\nEndmill DOC (mm) WOC (mm) RPM MRR ($cm^3/min$) Force (N) Power (W) 6mm 2F 6 3 11671 84.0 108.8 710 6mm 2F 4 3 11671 56.0 72.5 480 6mm 2F 2.5 2.5 11671 29.6 36.3 230 6mm 2F 6 0.6 11671 28.0 43.5 280 6mm 2F 6 2 11937 57.3 108.8 490 6mm 2F 6 0.4 11937 19.1 43.5 200 It is interesting that a 2.5mm/2.5mm DOC/WOC has a higher MRR at a lower cutting force compared to 6mm/0.6mm DOC/WOC.\nEndmill Deflection The CNCcookbook recommends a maximum deflection of 20% of chipload. At 0.02mm FPT, that is a deflection of $0.004$ mm. They also recommend 0.025mm as a general guideline.\nHSMAdvisor calculates 0.01mm tool deflection at the most aggressive cut of 6mm DOC 3mm WOC.\nThe Sienci LongMill has 0.2mm in y at 29.4N (Youtube).\nFEA Simulation Worst Case (110N) X-Axis Y-Axis X-axis: 0.020 mm Y-axis: 0.018 mm Nominal (75N) X-Axis Y-Axis X-axis: 0.013 mm Y-axis: 0.013 mm The V2 has 0.013 – 0.020 mm on x and 0.013 – 0.018 mm on y. Because I’m using “bonded” contact sets, the actual deflection is likely to be 2-3$\\times$ more than the simulation.\nThis means the deflection under nominal load (0.03 – 0.04 mm) will exceed the CNCcookbook guideline of 0.025 mm.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-09-cnc-mill-v2/","summary":"Design Goal (Quantified) After a few more weeks of research and reading up I have decided to use deflection at the spindle as my primary design goal. I plan to use depth of cut (DOC) + width of cut (WOC) to estimate the acceptable deflection.\nDepth of Cut DOC in aluminum ranges from 0.254mm/0.76mm/1.6mm (Shapeoko Wiki) to 2.54mm (Vince Fab) and 6.35mm (Shapeoko Forum).\nRecommendations for the Tormach PCNC1100 is also 6.","title":"CNC Mill V2"},{"content":"Tripteron The Tripteron is a 3 DoF parallel cartesian robot. It is decoupled, which means each axis can be controlled independently and isotropic, which means it\u0026rsquo;s dexterity is unchanged throughout the workspace.\nPractical Limitations But there are some serious obstacles to building a parallel CNC as a first machine. It needs many precise, high strength joints. Precise because any backlash gets magnified by the length of the chain. Strong because the length also results in a large moment.\nMisumi has a side mounting hinge that will fit on standard 20 $\\times$ 20 extrusion. They are 110 per set, for a total of 990 (the entire serial design cost 900). This definitely won\u0026rsquo;’\u0026rsquo;t be feasible unless I can machine them myself.\nInherent Advantages? It\u0026rsquo;s not clear if a parallel mechanism has any inherent advantage in a 3-axis design. The axes are normally split:\nXY table + Z spindle Y table + XZ spindle In either case two axes are well supported across their entire length, so only the last axis experiences any significant deflection. Compared to the parallel mechanism which has three chains which all experience deflection.\nIf we assume that the total length of each chain is the same as the serial design, then they must have a larger second/polar moment of area. If we stick to conventional extrusion profiles, the Tripteron will require more material to achieve the same stiffness.\nConclusion I might reexamine a parallel 2 DoF mechanism for the XY table in the future, after I have a working CNC mill. But for now it seems like my best bet is to stick with a conventional design, and maximize strength and rigidity.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-06-08-back-to-serial/","summary":"Tripteron The Tripteron is a 3 DoF parallel cartesian robot. It is decoupled, which means each axis can be controlled independently and isotropic, which means it\u0026rsquo;s dexterity is unchanged throughout the workspace.\nPractical Limitations But there are some serious obstacles to building a parallel CNC as a first machine. It needs many precise, high strength joints. Precise because any backlash gets magnified by the length of the chain. Strong because the length also results in a large moment.","title":"Back to Serial"},{"content":"Overview Notation $$ \\begin{aligned} \u0026amp;\\Theta \u0026amp;\u0026amp;: \\text{Joint velocities} \\cr \u0026amp;\\Theta_{a/p} \u0026amp;\u0026amp;: \\text{Active/Passive} \\cr \u0026amp;W \u0026amp;\u0026amp;: \\text{Twist} \\cr \u0026amp;V \u0026amp;\u0026amp;: \\text{Velocity of end-effector} \\cr \u0026amp; \\Omega \u0026amp;\u0026amp;: \\text{Angular Velocity} \\cr \u0026amp; X \u0026amp;\u0026amp;: \\text{Generalized coordinates} \\end{aligned} $$\nOutline Inverse Kinematics : Relates position\nInverse Jacobian : Relates velocities (in generalized coordinates)\nInverse Kinematic Jacobian : Relates velocities (in cartesian coordinates)\nInverse Kinematics Inverse kinematics are a way of getting joint coordinates from a desired end-effector pose. For a $3{\\text -}RPR$ planar robot it is given by\n$$ \\begin{cases} \\rho_1^2 = \u0026amp;x^2 + y^2 \\cr \\rho_2^2 = \u0026amp;(x+l_2 \\cos({\\Phi}) - c_2)^2 + (y+l_2 \\sin({\\Phi}))^2 \\cr \\rho_2^2 = \u0026amp;(x+l_3 \\cos({\\Phi + \\theta}) - c_3)^2 + (y+l_3 \\sin({\\Phi + \\theta}) - d_3)^2 \\end{cases} $$\nInverse Jacobian The joint coordinates and generalized coordinates are related by\n$$ A \\dot\\bold \\Theta_a + B\\dot\\bold X + C \\dot\\bold \\Theta_p = 0 $$\nFor the minimal kinematic set (MKS), where the joint velocities $\\dot X$ are reduced to the active joints $\\dot \\Theta_a$ , we can ignore $\\dot \\Theta_p$ and get\n$$ A \\dot \\bold \\Theta_a + B \\dot \\bold X = 0 $$\nIf $B^{-1}A$ is invertible,\n$$ \\dot \\Theta_a = \\underbrace{-A^{-1} B}_{J^{-1}}\\ \\dot\\bold X $$\nThe inverse jacobian is given by\n$$ J^{-1} = -A^{-1}B $$\nInverse Kinematic Jacobian The inverse kinematic jacobian $J_k^{-1}$ relates joint velocities to end-effector velocities. For the MKS\n$$ A \\dot \\bold \\Theta_a + B \\ \\underbrace{ H^{-1} \\bold W }_{\\dot\\bold X} = 0 $$\n$J_k$ is a transformation from generalized coordinates to cartesian twist.\nThe inverse kinematic jacobian is therefore\n$$ \\begin{aligned} J_k^{-1} \u0026amp;= -A^{-1} B H^{-1} \\cr \u0026amp;= \\enskip J^{-1} H^{-1} \\end{aligned} $$\nAcceleration Analysis Directly by differentiating\n$$ \\dot \\bold \\Theta = J_K^{-1} \\bold W $$\nto get\n$$ \\ddot \\Theta = J_k^{-1} \\dot \\bold W + \\dot J_K^{-1} \\bold W $$\nSummary Inverse Kinematic\n$$ \\begin{cases} \\rho_1^2 = \u0026amp;x^2 + y^2 \\cr \\rho_2^2 = \u0026amp;(x+l_2 \\cos({\\Phi}) - c_2)^2 + (y+l_2 \\sin({\\Phi}))^2 \\cr \\rho_2^2 = \u0026amp;(x+l_3 \\cos({\\Phi + \\theta}) - c_3)^2 + (y+l_3 \\sin({\\Phi + \\theta}) - d_3)^2 \\end{cases} $$\nInverse Kinematic Jacobian\n$$ \\begin{aligned} J_k^{-1} \u0026amp;= -A^{-1} B H^{-1} \\cr \u0026amp;= \\ \\ \\ J^{-1} H^{-1} \\end{aligned} $$\nAcceleration\n$$ \\ddot \\Theta = J_k^{-1} \\dot \\bold W + \\dot J_K^{-1} \\bold W $$\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-05-24-inverse-kinematics-velocity-acceleration-analysis/","summary":"Overview Notation $$ \\begin{aligned} \u0026amp;\\Theta \u0026amp;\u0026amp;: \\text{Joint velocities} \\cr \u0026amp;\\Theta_{a/p} \u0026amp;\u0026amp;: \\text{Active/Passive} \\cr \u0026amp;W \u0026amp;\u0026amp;: \\text{Twist} \\cr \u0026amp;V \u0026amp;\u0026amp;: \\text{Velocity of end-effector} \\cr \u0026amp; \\Omega \u0026amp;\u0026amp;: \\text{Angular Velocity} \\cr \u0026amp; X \u0026amp;\u0026amp;: \\text{Generalized coordinates} \\end{aligned} $$\nOutline Inverse Kinematics : Relates position\nInverse Jacobian : Relates velocities (in generalized coordinates)\nInverse Kinematic Jacobian : Relates velocities (in cartesian coordinates)\nInverse Kinematics Inverse kinematics are a way of getting joint coordinates from a desired end-effector pose.","title":"Inverse Kinematics, Velocity \u0026 Acceleration Analysis"},{"content":"Background Parallel manipulators are closed-loop kinematic chain mechanisms linked by several independent chains (Merlet, J. P. 2006). They have several advantages over serial manipulators such as accuracy and stiffness.\nBecause the chains are independent, each link does not have to support all subsequent links and can be lighter and smaller. The lower load and shorter link length reduce the (unmeasured) deformation in the system. The shorter kinematic chains also reduce the effect of deformation, actuator error and measurement error on the final end-effector position.\nIn a CNC machine In the case of a 3-axis CNC machine, at least 2 axes will be in series. In the previous post it is the y and z-axis. Any deformation in the y-axis is magnified by the length of the z-axis. Any forces on the spindle are also multiplied by the z-axis.\nStrength of Aluminum Extrusion Before delving into parallel mechanisms, I need to determine if rigidity and accuracy are a problem in the current design. I will start with the frame.\nA design based on aluminum extrusions will depend largely on the strength of the fasteners. The standard 2020 extrusion has a wall thickness of 2 mm, and a overhang of 3 mm. T-nuts have widths from 6 – 15 mm. This imposes a limit on the bolt tension before it starts to shear off the extrusion wall. The bolt tension and coefficient of friction (between aluminum and steel, not high) further limits the ability of joints to resist sliding.\nThe layout of each joint also affects it\u0026rsquo;s strength. An extrusion that lays on top of another extrusion is much stronger than a joint that depends only on friction. The final option is to drill a hole, and bolt straight through the extrusion.\nDeformation and Accuracy Based on the FEA simulation, we saw a deflection of 0.8 – 1.6 mm with 10 N of force. A 500 W spindle cutting aluminum is capable of generating around 30 – 60 N of force on the frame (at higher depth of cut). The simulation also overestimated the fastener joints.\nThis means that in the current design, frame rigidity is a major bottleneck in cutting metal. One obvious solution is to change the material and/or method of construction. This would definitely increase the cost and amount of fabrication required.\nI think that I will stick to aluminum extrusions because of its low cost, availability and the sheer convenience of being able to bolt a frame together. My hope is that a parallel mechanism design will overcome the material limitations.\nPlanar Parallel Mechanism I think the most obvious starting point is to have a spindle mounted on a moving z-axis, and a planar parallel mechanism to replace an x-y table. One of the most common, and well-studied mechanisms is the 3-RPR robot.\nThe notation reads as: 3 independent legs with revolute – prismatic – revolute joints per leg. Each leg will have one active (driven) joint and two passive joints, most commonly the prismatic joint.\nMATLAB Simulation (and visualization) I started by writing a quick kinematic simulation in MATLAB. The base are denoted by blue circles, platform by red circles and legs by black lines. The grey circles show the maximum extension of each prismatic joint. There is no limit on the range for the revolute joints.\nIn subsequent posts I will explore\nThe workspace Which are the locations that can be reached Singularities Which are poses where the mechanism loses controllability and/or rigidity Velocity, acceleration and static analysis Used to determine the relation between joint velocities/force and end-effector velocity/force ","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-05-23-parallel-manipulators/","summary":"Background Parallel manipulators are closed-loop kinematic chain mechanisms linked by several independent chains (Merlet, J. P. 2006). They have several advantages over serial manipulators such as accuracy and stiffness.\nBecause the chains are independent, each link does not have to support all subsequent links and can be lighter and smaller. The lower load and shorter link length reduce the (unmeasured) deformation in the system. The shorter kinematic chains also reduce the effect of deformation, actuator error and measurement error on the final end-effector position.","title":"Parallel Manipulators"},{"content":"Overview I’ve been looking at getting a CNC mill to machine small, strong, metal parts. Initially I looked at the chinese mills such as the 3018 ($400). But they have a reputation of not lasting very long. Then I moved on to the Shapeoko 3 ($1550) and OpenBuilds C-Beam ($1200), but they’re too big to fit comfortably in an apartment. The Carbide3D Nomad ($2750) would be perfect. It is rigid, compact and easily enclosed.\nI decided that this would be a great chance to learn and design my own CNC mill.\nDesign Objectives Rigid, to cut aluminium Enclosed and quiet Minimal fabrication, using as many off-the-shelf parts as possible Compact, to fit in an apartment Key Parameters Work area: 200mm x 200mm x 100mm HGR20 Linear Rails 2x HGR20 linear rails for each axis SFU1204 Ballscrews with NEMA 23 1500W air-cooled Spindle Summary Aluminum profiles bolted together Linear rails and ball screws for all axes Finite Element Analysis I used FEA simulation to get an idea of the strain and deformation that I will see in the frame. The fact that the extrusions are bolted together complicates things, because it cannot be modeled as one large, ‘bonded’ component.\nTo approximate the fasteners, I set\nThe extrusions to not interact with each other The gussets and connectors to ‘bonded’ with every other component X-Axis Y-Axis Simulations were done along the x and y axes with a force of 10N.\n","permalink":"https://andrelimzs.github.io/posts/cnc-mill/2020-05-23-designing-a-cnc-mill/","summary":"Overview I’ve been looking at getting a CNC mill to machine small, strong, metal parts. Initially I looked at the chinese mills such as the 3018 ($400). But they have a reputation of not lasting very long. Then I moved on to the Shapeoko 3 ($1550) and OpenBuilds C-Beam ($1200), but they’re too big to fit comfortably in an apartment. The Carbide3D Nomad ($2750) would be perfect. It is rigid, compact and easily enclosed.","title":"Designing a CNC Mill"}]