The Giant's Castle is the primary residence of The Giant from the fairy tale, "Jack and the Beanstalk."
The last of the fairy tales to be real by the Ethereal Solidifier after Slimer hid a book under it, was a giant beanstalk. It grew out out the read outside the Firehouse into the sky above. The Giant's Castle then materialized in the clouds. Peter Venkman and Winston Zeddemore took Ecto-2 up to investigate giant trash raining down from the castle.
They were overpowered by The Giant's Son and taken into the castle. Peter and Winston were presented to the Giant, who was in the middle of eating in the kitchen/dining room. Once the Solidifer's effect was reversed, the castle (along with everything else from the tale) dematerialized.