Critical points with a degenerate Hessian and direct study of the sign

The only critical point is the origin. The determinant of the Hessian is zero, hence the criterion does not decide and we study directly the sign of the difference $f(x,y)-f(0,0)$. The difference is the sum of the two terms $x^2$ and $y^4$, both nonnegative, therefore it is $\ge 0$ on the whole plane and the origin is an absolute minimum point, with value $0$. The minimum is also strict, since the sum vanishes only when $x=0$ and $y=0$ simultaneously.

Here too the only critical point is the origin and the Hessian is degenerate, with the same matrix as in the previous exercise: the criterion does not distinguish the two situations, which are in fact different. For the direct study we look at the restriction to the $y$-axis, which equals $y^3$ and changes sign across the origin. Hence in every neighborhood of the origin the function takes values both greater and smaller than $f(0,0)=0$, therefore the origin is a saddle point. To rule out an extremum it is sufficient to have one restriction on which the sign of the difference changes.

Expanding the product we obtain $f(x,y)=y^2-5x^{2}y+4x^4$ and the system is solved by substitution: from the second equation $y=\frac{5}{2}{x}^2$, and the first becomes $-9x^3=0$, hence the only critical point is the origin, with a degenerate Hessian. The restriction to any line through the origin has positive sign near $0$: on $y=mx$ with $m\ne 0$ we have $f(x,mx)=x^2(m-x)(m-4x)$, which for small $x$ behaves like $m^2{x}^2>0$, and on the $x$-axis we have $f(x,0)=4x^4>0$. The lines alone would suggest a minimum, but the conclusion is different: on the parabola $y=2x^2$, which lies between the two parabolas $y=x^2$ and $y=4x^2$ where the two factors have opposite signs, the function equals $-2x^4<0$, while on the $y$-axis it equals $y^2>0$. Hence in every neighborhood of the origin the difference $f(x,y)-f(0,0)$ takes both signs, therefore the origin is a saddle point.