You should always post code which you have tested. Your posted example doesn't work because you don't use InitSprite() at the beginning (an error message is displayed which gives you that hint) and your code doesn't have any event loop so the program ends at once. In order to answer your question I first had to do this for you:
Code:
InitSprite()
OpenWindow(0, 0, 0, 800, 600, "")
OpenWindowedScreen(WindowID(0), 100, 100, 600, 400, 0, 0, 0)
SetWindowColor(0, RGB(255,0,0))
Repeat
Until WaitWindowEvent() = #PB_Event_CloseWindow
To set the window background color you have two options:
1. Set the subsystem to Gtk2 (Menu > Compiler > Compiler Options... > Library Subsystem: gtk2)
2. When using the default Gtk3, trying to use SetWindowColor() will most likely fail in most Linux distributions. This is not PureBasic's fault but that of Gtk3. During the development of Gtk3 at least 3 times the required methods to change the background color have been modified so being a moving target for PureBasic. But you may try
this workaround which was tested successfully on 16 different Linux versions with different Gtk3 versions.