Why doesn't it work properly to render the same scene graph in two different windows?

The reason is probably that the two windows do not share the same GL context, but the render actions are not set up with different cache contexts.

Use two different values with SoGLRenderAction::setCacheContext() for the two windows' render actions.

  for (int i=0; i < NR_VIEWERS; i++) {
    SoGLRenderAction * renderaction = window[i]->getGLRenderAction();
#ifdef __COIN__
  /* In Coin, we have added an extension to SoGLCacheContextElement
     for getting a _guaranteed_ unique number for the cache contexts.
     That the original Inventor API is missing a function like this
     should be considered an ugly flaw in the design. */
    uint32_t uniqcc = SoGLCacheContextElement::getUniqueCacheContext();
#else // SGI or TGS Inventor
  /* Use a "pseudo-random" number and cross your fingers that it's
     unique, since there's no decent method in the API from which
     the application programmer can get a unique cache context id. */
    uint32_t uniqcc = 19720408 + i;
#endif // SGI / TGS Inventor
    renderaction->setCacheContext(uniqcc);
  }

This is taken care of automatically by Systems in Motion's SoQt / SoGtk / SoXt / SoWin / Sc21 libraries, so you shouldn't have to worry about it unless you are making your own GUI binding to Coin from scratch.