I'll give you a road map. A complete solution seems to need some work.
I would take the derivative of $ f $, and then find the critical points, where the derivative doesn't exist or equals zero, and finally compare the values of $ f $ at the critical points, and those at the interval boundary points, to find the maximum.
1. Find critical points
Take the derivative of $ f $, which can be done manually.
1.1 Derivative doesn't exist
The derivative doesn't exist at $ x = 1 $, which is also a boundary point.
1.2 Derivative is zero
To solve $ f'(x) = 0 $, I came up with a fourth degree equation with coefficients in terms of $ a $ and $ b $:
$$
4a^2x^4 + 4abx^3 + (-4a^2 + b^2 + 1)x^2 - 4abx - b^2 = 0
$$
You can find an analytic solution for this using the root formulas for the quartic equation, or using a computer algebra system.
I would use SymPy because I am already familiar with Python. SageMath is another good option. There is of course others.
The following Python code can be run in a SageMath Cell:
from sympy import Symbol
a = Symbol('a')
b = Symbol('b')
solve(4*a**2*x**4+4*a*b*x**3+(-4*a**2+b**2+1)*x**2-4*a*b*x-b**2==0, x)
After you have an expression in terms of $ a $ and $ b $ for $x$ values that make the derivative zero, you will need to put that expression in this inequality $ 0 \leqslant x < 1 $ to find values of $ a $ and $ b $ that generate $ x $ values between $ 0 $ and $ 1 $. You can probably use a symbolic solver for this, too.
2. Compare values of $ f $ at critical points
Compare values of $ f $ at critical points that you have found above to find the largest.