Jarrod Bowen gave Hull the lead when he got past Michael Mancienne.
Soon after, Nouha Dicko put the hosts 2-0 up from close range, whilst Mustapha Carayol missed the chance to draw back he headed wide just minutes later.
Apostolos Vellios did get a brilliant finish late on from Matty Cash’s cross, but it was not enough.
Adkins’ side had the chance to make an early lead when Marcus Henriksen’s free-kick forced a save from Forest goalkeeper.
Just before he gave the hosts a 1-0 lead, Bowen almost scored an outrageous lob from 40 yards when he caught Smith off his line, but his effort ultimately cannoned off the post.
Bowen has scored 12 goals in all competitions for Hull City so far this term, which is twice as many goals scored as any of his teammates.
After Dicko extended the hosts’ lead, he came close to finding his second and Hull’s third just after the break when he fired from outside the box and the ball rebounded off Smith and out of play just before it ran into the path of Michael Hector.
Hull keeper David Marshall pulled off an incredible save with 15 minutes left when he tipped Joe Worrall’s powerful header from inside the box over the bar.
But Forest couldn’t replicate the form that saw them knock out Arsenal in the previous round.
Hull City boss Nigel Adkins said
“I’m pleased we won. it was well deserved. We had a lot of opportunities again and I’m pleased about that. I’m happy we took two goals.
“For me it was a good performance from the players but as a collective team there was a lot of spirit about everybody.
“We created a lot of opportunities to win the game and the supporters were right behind everybody as well.”
Forest manager said
“I think it was clear that we came here thinking it was going to be an easy game. We can’t forget that most of Hull’s players are Premier League players.
“Hull showed that they were much more mature. We need to improve and we need to grow, especially in personality. We need to learn, we need to keep going. Stupid mistakes cost us the game.
“When you are not experienced enough, you think that because you have won one important game, the following game is going to be easy. I told them before the game but you can learn when you are more mature.”