Nanofluids have been shown experimentally to have high thermal conductivity. In this study, the convective instabilities in a horizontal viscoelastic nanofluid saturated by porous layer under the influences of gravity and magnetic field are investigated. The linear stability theory is used for the transformation of the partial differential equations to system of ordinary differential equations through infinitesimal perturbations, scaling, linearization and method of normal modes with two-dimensional periodic waves. The system is solved analytically for the closed form solution of the thermal Darcy-Rayleigh number by using the Galerkin-type weighted residuals method to investigate the onset of both stationary and oscillatory convection. The effects of the scaled stress relaxation parameter, scaled strain retardation parameter and Chandrasekhar number on the stability of the system are investigated. The scaled strain retardation parameter stabilizes while the scaled stress relaxation parameter destabilizes the nanofluid system. The system in the presence of magnetic field is more stable than the system in the absence of magnetic field.