(x2+y)2-10(x2+y)+16

Let u=x2+y. Substitute u for all occurrences of x2+y.

u2-10u+16

Consider the form x2+bx+c. Find a pair of integers whose product is c and whose sum is b. In this case, whose product is 16 and whose sum is -10.

-8,-2

Write the factored form using these integers.

(u-8)(u-2)

(u-8)(u-2)

Replace all occurrences of u with x2+y.

(x2+y-8)(x2+y-2)

Factor (x^2+y)^2-10(x^2+y)+16