Robot-assisted surgery is gaining popularity worldwide and there is increasing scientific interest to explore the potential of soft continuum robots for minimally invasive surgery. However, the remote control of soft robots is much more challenging compared with their rigid counterparts. Accurate modeling of manipulator dynamics is vital to remotely control the diverse movement configurations and is particularly important for safe interaction with the operating environment. However, current dynamic models applied to soft manipulator systems are simplistic and empirical, which restricts the full potential of the new soft robots technology. Therefore, this article provides a new insight into the development of a nonlinear dynamic model for a soft continuum manipulator based on a material model. The continuum manipulator used in this study is treated as a composite material and a modified nonlinear Kelvin-Voigt material model is utilized to embody the visco-hyperelastic dynamics of soft silicone. The Lagrangian approach is applied to derive the equation of motion of the manipulator. Simulation and experimental results prove that this material modeling approach sufficiently captures the nonlinear time- and rate-dependent behavior of a soft manipulator. Material model-based closed-loop trajectory control was implemented to further validate the feasibility of the derived model and increase the performance of the overall system.