If you think that Google isn’t serious about ranking non-responsive (a.k.a. not-mobile-ready) websites lower than those that are truly mobile and tablet ready, think again. They are serious enough to publish specific tips and tricks.
Changing over to a responsive site is, unfortunately, a lot of work but think of it as just a “necessary upgrade.” Sure, you can take the not-to-worry approach, but all that means is that your ranking on Google will go down, and non-desktop visitors to your site will not stay as long, if at all. After all, why bother to do all the scrolling and zooming when other sites are easier to read.
More information, along with Google’s tips and tricks can be found here: https://www.seroundtable.com/google-static-responsive-design-20033.html (there are many, many other sites with the same information).