Build could have been far worse (just look at Zi-O), but Drive definitely had a lot of problems that pulled it down, with only a few saving graces. I don't speak for everyone, but here was what I personally got out of the show:
The Good
1. Chase - Pretty much the only character I really liked in the show. It was pretty sad that the one character who was supposed to have no emotions happened to also be the only one with a personality. He's what got me through to the final arc.
2. The villains - I really love it when a show develops the characters from all sides, and that goes a long way towards my being able to like a show. In terms of Drive, Chase started off as an interesting (if cliche) villain, then switched sides. Brain had a lot of potential but was never developed. But in the final arc, you actually FEEL for the the villains. It was too little too late, but I have to give them some credit for making viewers want to side with the "bad guys" right before they're literally rendered extinct.
3. I love detective themes. Agito was my first full Rider series, so I was happy that the show was formatted to be a detective series, although they didn't do it nearly as well as Agito or W.
The Bad
1. Krim Steinbelt - It's kind of interesting that they did a play on the popular tech brand Klim (I'm using a Klim keyboard now, in fact), paired with Frankenstein to come up with his name. The design of the driver was obviously meant for younger kids and reminded me a bit of Buddy Phones. But it started a trend of referring to Drivers as belts and Belt-san got pretty annoying at times with his cheap car sound reactions. Kiva did much better with the live Driver idea, and I think Krim should have been thought out better.
2. The car theme just didn't really work for a motorcycle-centric franchise and while I understand why they did it, the entire point fell apart when EVERY OTHER RIDER ended up with (or becoming) a Mashin. Perhaps if they'd gone all the way, Tridoron wouldn't have felt so out of place and useless.
The Ugly
1. Accel literally existed because of an in-joke left over from Kabuto. They did a much better take on the Rider = Mashin in Ex-Aid, right down to the serious/gloomy personality. Between Accel and Mach, the secondary riders were mostly just annoying.
2. The Cast - The majority of the cast had zero personality or depth, and the few they wanted to have "personality" ended up just being annoying (like Mach). In the end, it was the first series outside of Hibiki I wanted to walk away from, and I ONLY pushed myself through because of Chase. In fact, Chase is also pretty much the only thing that made it better than Ghost.
3. Plot Whiplash - I'm a professional writer. It's what I do, it pays my rent, and it's a lifelong passion. When I watch Kamen Rider, I usually just sit back and enjoy it for what it is - a kid's show. But Drive went beyond the normal flaws of a kid's series, and I think Zi-O's biggest downfall was that it tried to beat out Drive for the number of unnecessary plot twists and self-retcons. I remember a lot of the complaints in the threads during that series were how a major reveal would almost always be retconned within a couple episodes. Those changes for the sake of change often broke immersion in ways that make Saber look like a masterpiece.
Now, I know some people absolutely loved Drive, but it shared most of the problems that made Hibiki and Ghost so terrible without the excuse of losing its writers. It tried to overcomplicate things, failed to bring any depth to the cast, relied heavily on a Rider theme that was basically ignored by the rest of the Riders so Drive ended up looking like he was guest-starring in his own show, ended the series by making the good guys turn out to be the bad guys and making you sympathise completely with Heart and his mission... I can give a lot of excuses for Kamen Rider, but having seen much of the Showa Era at this point as well as all of Heisei to present (Hibiki's the only KR series I gave up on partway through), Drive just missed the mark way too often.
There are some things that weren't bad, and I'd probably sit through it again if REALLY bored, but a lot of us didn't like Drive for these (and other) reasons. And unlike Ghost, I wouldn't be interested in seeing the cast brought back for a remake, because I can't pretend the shoddy writing and acting was due to staffing issues. At least the Ghost cast gave it their all for as long as they could before they visibly gave up.