I had very similar setup at the time I bought my BMW.
Building boost was never an issue with the TD04, even at 2000rpm it could build 0.5 bars or 1 bars or anything I set.
-Actually it was hard to set, because of permanent boost creep. Previous owner even drilled a ~15mm hole next to the internal wastegate, without real success
It never dropped the boost at high rpm with that glowing red "plug" on the exhaust side, but power level was close (or below) the stock n/a engine.