Baker's free-kick handed the Sky Blues a 35th-minute lead but Leon Clarke's brace, the first from the penalty spot just before half-time, clinched all three points for the Iron.
Coventry have now lost 19 points from winning positions this season - the worst record in all four top divisions in England.
"I'd rather not score if it meant we were going to win but it didn't happen for us," Baker told City's official website. "It is all about the three points for me and that's what I focus on, not getting a goal.
"I think we threw away the game a little if I am honest. After we went 1-0 up, we conceded an awful goal. We could have stopped that easily by doing the right things.
"I didn't know whether it was a penalty or not, didn't look like it from where I was. I thought Clarke had handled it. The ref has given it and it has let them back in the game when we could have been out of sight really. We haven't got going in the second half at all and that was really disappointing. We played well when we went behind but that isn't the way it should be as we have to keep up that tempo which is required for the whole game."
