The GegenbauerC(n, a, x) function computes the nth Gegenbauer polynomial - see Abramowitz and Stegun, Handbook of Mathematical Functions, Chap. 22.

When all of $\left\{2a\,1+n\,n+2a\right\}$ are not a negative integer or zero, the Gegenbauer polynomials satisfy:

GegenbauerC(n,a,z) = 'piecewise'(n::negint,0, n=0, 1,convert(GegenbauerC(n,a,z),hypergeom));

$\mathrm{GegenbauerC}\left(n\,a\,z\right)=\left\{\begin{array}{cc}0& n::{\mathbb{Z}}^{-}\\ 1& n=0\\ \frac{\mathrm{\Gamma }\left(n+2a\right)\mathrm{hypergeom}\left(\left[-n\,n+2a\right]\,\left[\frac{1}{2}+a\right]\,-\frac{z}{2}+\frac{1}{2}\right)}{\mathrm{\Gamma }\left(1+n\right)\mathrm{\Gamma }\left(2a\right)}& \mathrm{otherwise}\end{array}\right.$

and are orthogonal on the interval $\left[\mathrm{-1}\,1\right]$ with respect to the weight function $w\left(z\right)={\left(-z^2+1\right)}^{a-\frac{1}{2}}$:

Int(w(z)* GegenbauerC(m, a, z) * GegenbauerC(n, a, z), z=-1..1) = 'piecewise'(n=m, Pi*2^(1-2*a)*GAMMA(n+2*a)/(n!*(n+a)*GAMMA(a)^2),0);

${\int }_{\mathrm{-1}}^{1}w\left(z\right)\mathrm{GegenbauerC}\left(m\,a\,z\right)\mathrm{GegenbauerC}\left(n\,a\,z\right)\ⅆz=\left\{\begin{array}{cc}\frac{\mathrm{\pi }{2}^{1-2a}\mathrm{\Gamma }\left(n+2a\right)}{n!\left(n+a\right){\mathrm{\Gamma }\left(a\right)}^2}& n=m\\ 0& \mathrm{otherwise}\end{array}\right.$

When any of $\left\{2a\,1+n\,n+2a\right\}$ is a negative integer or zero, the Gegenbauer polynomials are computed using the following identity:

GegenbauerC(n,a,z) = (2*a*z*(1+2*a)*GegenbauerC(n-1,1+a,z) + 4*(-1+z^2)*a*(1+a)*GegenbauerC(n-2,a+2,z)) / ((n+2*a)*n);

$\mathrm{GegenbauerC}\left(n\,a\,z\right)=\frac{2az\left(1+2a\right)\mathrm{GegenbauerC}\left(n-1\,1+a\,z\right)+4\left(z^2-1\right)a\left(1+a\right)\mathrm{GegenbauerC}\left(n-2\,a+2\,z\right)}{\left(n+2a\right)n}$

which in turn can be derived from the differential equation with respect to z satisfied by this function:

f(z) = GegenbauerC(a,b,z);

$f\left(z\right)=\mathrm{GegenbauerC}\left(a\,b\,z\right)$

diff(f(z),z,z) = (-1-2*b)*z/(-1+z^2)*diff(f(z),z)+a*(2*b+a)/(-1+z^2)*f(z);

$\frac{{\ⅆ}^2}{\ⅆz^2}f\left(z\right)=\frac{\left(-1-2b\right)z\left(\frac{\ⅆ}{\ⅆz}f\left(z\right)\right)}{z^2-1}+\frac{a\left(2b+a\right)f\left(z\right)}{z^2-1}$

For n::posint and n > 1 and a <> 0, the Gegenbauer polynomials satisfy the following recurrence relations:

GegenbauerC(0,a,z) = 1:

GegenbauerC(1,a,z) = 2*a*z:

GegenbauerC(n,a,z) = 2*(n+a-1)/n*z*GegenbauerC(n-1,a,z) - (n+2*a-2)/n*GegenbauerC(n-2,a,z):

and for a = 0, they are related to the ChebyshevT polynomials:

GegenbauerC(n,0,z) = 2/n*ChebyshevT(n,z):

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}n::\mathrm{negint}$

$0$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}n=0$

$1$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(3\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)$

$\frac{4a\left(z^{2}a+2z^2-\frac{3}{2}\right)z\left(1+a\right)}{3}$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}a::\mathrm{negint}$

$0$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(2\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}a=0$

$2z^2-1$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;-z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}a=0,n::\mathrm{posint}$

${\left(\mathrm{-1}\right)}^n\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}z=0$

$\frac{2^n\mathrm{\Gamma }\left(a+\frac{n}{2}\right)\sqrt{\mathrm{\pi }}}{\mathrm{\Gamma }\left(a\right)\mathrm{\Gamma }\left(\frac{1}{2}-\frac{n}{2}\right)\mathrm{\Gamma }\left(1+n\right)}$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}z=1,n::\mathrm{nonnegint}$

$\frac{\mathrm{\Gamma }\left(n+2a\right)}{\mathrm{\Gamma }\left(1+n\right)\mathrm{\Gamma }\left(2a\right)}$

$\mathrm{simplify}\left(\mathrm{GegenbauerC}\left(n\&comma;a\&comma;z\right)\&comma;'\mathrm{GegenbauerC}'\right)\mathbf{assuming}z=-1,n::\mathrm{nonnegint}$

$\frac{{\left(\mathrm{-1}\right)}^n\mathrm{\Gamma }\left(n+2a\right)}{\mathrm{\Gamma }\left(2a\right)n!}$