Modeling neural circuits with detailed biophysical properties requires novel optimization toolkits to enable scalability. In this study, we develop a hybrid method based on random-initialization of stochastic gradient descent algorithm to tune the parameters of neural circuit models, for a given objective in a supervised learning setting. We present ODYNN, an optimization suite for dynamic neural networks. ODYNN is designed for simulating and optimizing biological neural circuits to model multi-scale behaviors exhibited by the neural dynamics, and to test neuroscience related hypotheses. It is enhanced with features such as performing experiments with different biophysically realistic neuronal models as well as artificial recurrent neural networks (in particular Long Short Term Memory), incorporating calcium imaging data, to design arbitrarily structured neural circuits and optimizing them to govern specific behaviors. For a given neural circuit structure, a given neuronal and synaptic model, ODYNN adopts a random search optimization step followed by an adaptive-momentum gradient-based learning algorithm to learn the calcium imaging data of desired input/output dynamics. We perform single neuron optimizations for biophysically realistic complex neuron models which are realized by partial differential equations, and explore in their parameter space to reason about neuronal dynamics at cell level resolution. We then perform synaptic and neural parameter optimization in small neuronal circuits, by the hybrid optimization approach and assess the performance by learning models to express arbitrarily chosen dynamics. Furthermore, we demonstrate that ODYNN is able to scale up to the tuning of larger neural circuits' parameter spaces, efficiently, in a reasonable optimization duration. We optimize neural circuits such as the tap-withdrawal, a neural system from the nervous system of the soil worm, C. elegans, that induces reflexive response as a result of mechanical input stimulations; and also the forward locomotion circuit of the C. elegans nematode which is responsible for generating traveling waves into the muscle cells to generate the worm's forward crawling.