Can this notation be defined: with and if yes its value will be ?
Alternatively, can be a factorial of a negative number and when ?
The definition of is not .
The definition is the number of subsets with size k of a set of size n. It appears that if , then .
So yes, when because there is no subset of size k of a set of size n in this case.
For the second question, if factorial is defined by , then we have . But it's not really usual to use factorial on negative integers (beside 0).
(But you can expand the fatorial to with the Euler's map.)
We can define
for and (or , complex numbers)
if and follows from that definition that .
With that definition we can prove the identity
we can use this identity to extend the definition of the binomial for n<0,
for example, put n=0
so
by induction we can prove