When working an optimization problem, when should I consider the domain of the primary equation?

Also, when it is determined that the interval of which the variable is an element is open, do I consider the endpoints? For example. Let's say that

is continuous on the open interval

and has a relative maximum at

. But at the point

, with

,

. This is strange because, when I differentiate with respect to the variable and find all values for which

, all I get is the relative extrema, And I can't consider the endpoints because they are not in the domain. What do I do?