The divergence of F:

$\nabla ·\mathbf{F}\={\partial }_x\left(x y^3\right)\+{\partial }_y\left(y z\right)\+{\partial }_z\left(x^{2}z\right)\=y^3\+z\+x^2$ 

Implement the integral of $\nabla ·\mathbf{F}$ over the interior of $R$:

${\int }_{-\sqrt{5\/3}}^{{\sqrt{5\/3}}_{}}{\int }_{-\sqrt{\left(5-3 x^2\right)\/2}}^{\sqrt{\left(5-3 x^2\right)\/2}}{\int }_0^{5-3 x^2-2 y^2}\mathrm{divF} \mathrm{dz} \mathrm{dy} \mathrm{dx}$ = $\frac{875}{216}\sqrt{6}\mathrm{\π}$

To compute the flux through $R$, note that there are two boundaries, the upper elliptic paraboloid, and the ellipse and its interior that is the intersection of the paraboloid and the plane $z\=0$. To compute the flux through the upper surface, note that on that surface

If this be integrated over the ellipse $3 x^2\+2 y^2\=5$, the result is

${\int }_{-\sqrt{5\/3}}^{{\sqrt{5\/3}}_{}}{\int }_{-\sqrt{\left(5-3 x^2\right)\/2}}^{\sqrt{\left(5-3 x^2\right)\/2}}\left(6 x^2{y}^3\+\left(4 y^2\+x^2\right) \left(5-3 x^2-2 y^2\right)\right) \mathrm{dy} \mathrm{dx}$ = $\frac{875}{216}\sqrt{6}\mathrm{\π}$ 

On the lower boundary (ellipse), the outward normal is $\mathbf{N}\=-\mathbf{k}$, so $\mathbf{F}·\mathbf{N}\=-x^{2}z$, which becomes zero in the plane $z\=0$. Hence, the flux through the interior of the ellipse at the bottom of $R$ vanishes, and the flux through the upper surface matches the volume integral of the divergence.

The Student VectorCalculus package is needed for calculating the divergence, but it then conflicts with any multidimensional integral set from the Calculus palette. Hence, the Student MultivariateCalculus package is installed to gain Context Panel access to the MultiInt command.

There are two parts to the boundary of $R$, the surface of the upper hemisphere, and, in the plane $z\=0$, the interior and boundary of the ellipse $3 x^2\+2 y^2\=5$. For the flux through the upper surface, use a task template.

On the lower boundary (ellipse), the outward normal is $\mathbf{N}\=-\mathbf{k}$, so $\mathbf{F}·\mathbf{N}\=-x^{2}z$, which becomes zero in the plane $z\=0$. Hence, the flux through the interior of the ellipse at the bottom of $R$ vanishes, and the flux through the upper surface matches the volume integral of the divergence.

On the lower boundary (ellipse), the outward normal is $\mathbf{N}\=-\mathbf{k}$, so $\mathbf{F}·\mathbf{N}\=-x^{2}z$, which becomes zero in the plane $z\=0$. Hence, the flux through the interior of the ellipse at the bottom of $R$ vanishes, and the flux through the upper surface matches the volume integral of the divergence.