We construct new many-body invariants for 2D Chern and 3D chiral hinge insulators characterizing quantized pumping of bulk dipole and quadrupole moments. The many-body invariants are written entirely in terms of many-body ground state wave functions on a torus geometry with twisted boundary conditions and a set of unitary operators. We present a number of supporting arguments for the invariants via topological field theory interpretation, adiabatic pumping argument, and direct mapping to free-fermion band indices. Therefore, the invariants explicitly encircle several different pillars of theoretical descriptions of topological phases. Furthermore, our many-body invariants are written in forms which can be directly employed in various numerics including the exact diagonalization and the density-matrix renormalization group simulations. We finally confirm our invariants by numerical computations including an infinite density-matrix renormalization group on quasi-one-dimensional systems.