I'm not sure about GTK3. But I know that KDE plasma has a setting for legacy X11 applications (I think all this fractional scaling stuff requires wayland). It can render them as-is or scale them up (bitmap scaling).