Kurt Wolff had already gained experience in the publishing industry in Germany with his own publishing house, Kurt Wolff Verlag, which he had run from 1913. Wolff was engaged in supporting modern and expressionist German and foreign literature, as well as Jewish writers and art. His interest in art led him to found in 1923 Pantheon Casa Editrice in Florence, Italy, together with his partner Giovanni Mardersteig, publishing high quality art books in cooperation with renowned scholars and enjoying international distribution. The financial crisis of 1929 and the political climate in Germany forced Kurt Wolff to close his publishing houses in 1930 and emigrate to Paris. During the 1930s he lived in Nice and Paris with his second wife, Helen Wolff (1906–1994). After a period of internment and with the help of Varian Fry of the European Rescue Committee, the couple were able to emigrate in 1941 to New York. Within a year they re-established themselves in the publishing business and, in 1942, with financial help from friends, set up Pantheon Books in their apartment at 41, Washington Square.

The company focused on fine editions by German and French authors, either in the original or in English translation. One of Pantheon's first books was Meine Wanderungen, a book of poetry by Ivan Heilbut. For the French translations they relied on the German émigré Jaques Schiffrin, who translated books by Louis Aragon, André Gide and Joseph Kessel. As the emerging publishing market was hard to break into, Kurt Wolff searched for a publishing niche and German authors who had not been translated for the American market. The books he chose included the fairy tales of Gebrüder Grimm (illustrated by the German émigré artist Josef Scharl), as well as books by Jacob Burckhardt and Erich Kahler, among others. In 1942 the German émigré Ivan Heilbutt published Francisco and Elizabeth, illustrated by Raffael Busoni. It told the story of a friendship between a seven-year-old girl called Elizabeth and a small refugee boy, Francisco, who emigrate with the boy's mother to New York City to escape the Spanish Civil War (Philipps 2001, 100).

Besides literature, Pantheon's list included books on art and photography, as well as on the humanities and psychology. In 1947, the German émigré photographer Fred Stein published his 5th Avenue photobook with Pantheon. His first photobook, it was received with high praise, as can be seen in the reviews, and enhanced his professional reputation. The photobook starts at the south end of 5th Avenue and, over the course of 100 images, follows it north, to end in Harlem. The images capture well the diversity and changing character of the street and the variety of cultures, ethnicities and social conditions of the inhabitants can be seen as a metaphor for the city. Fred Stein was a sensitive photographer, his approach always humanistic and touched with humour as he captured life on the streets. Employing unusual perspectives, angles and details of everyday scenes, the book is dedicated to the historic street and captures the essence of metropolitan life through a mix of street views, architectural shots and pictures of people. Although the narrative of the book is provided by the walk from the south to the north end of 5th Avenue, the careful selection and arrangement of the images add to its impact. Two images on a double-page spread enter into a dialogue with one another and provide interesting juxtapositions as well as commonalities, so that flicking through the pages allows the reader to share Stein's experience of walking down the street. The same year, 1947, a French edition of 5th Avenue was published with the help of the émigré publishing house Querido, co-founded by the German émigré publisher Fritz H. Landshoff (the great uncle of the émigré photographer Hermann Landshoff). Both founders, Kurt Wolff and Fritz H. Landshoff, were photographed by Fred Stein for his series on émigré authors and intellectuals in exile in New York.

Other photobooks with the topic on New York published by émigré photographers include: Manhattan Magic by Mario Bucovich, Chinatown U.S.A. by Elizabeth Coleman, Times Square by Rolf Tietgens, New York by Andreas Feininger as well as a collection of essays and images by Ann Tizian Leitich in her book New York.
The photobook had several functions for the émigré photographer. As a publishing medium it was independent of the art market and museums and favoured self-representation (Parr 2004, 9–11; Ziehe 2015, 9). It also allowed to secure the self-marketing, economic and financial conditions for emigrated photographers (Seeber 2015, 10–13), and offered the photographer an opportunity to build a professional reputation in exile. Furthermore, photographs taken before emigration could be brought before a wider public. (Grebe 2015, 171). Pantheon was not the only publishing house to support the creation of photobooks by émigré photographers. Among the others who did so were: J.J. Augustin (Horst P. Horst, George Hoyningen-Huene, André Kertész, Alexey Brodovitch); Oceana Publications (Marion Palfi) and Schocken Books (Roman Vishniac). Other émigré photographers who published their work in book form were Ylla, Lilo Hess as well as Henry Rox.

Starting in 1956 the émigré photographer Tim Gidal published together with his wife, Sonia Gidal, a youth literature series My village in … , with Pantheon. The two firsts books were My village in Austria (Pantheon Books, 1956) and My village in India (Pantheon Books,1956). Between 1956 and 1970 25 volumes were published covering villages around the world. All books had the same structure. On the front endpapers was drawn a detailed plan of the village where the "hero" of the story lives. Then it followed an instructive account of life in that village. The text was written by Sonia Gidal, while Tim Gidal was responsible for the photographs. After living in several countries in exile, the series can be seen as the photographic output and human living experience of the nomadic life of Tim and Sonia Gidal. Parallel to Pantheon, the publishing house Orell Füssli in Zurich started in 1961 with the German edition of the youth series Mein Dorf in …  and until 1968 a total of 10 volumes were published.

In 1946/47 Pantheon Books moved to bigger premises at 333 6th Avenue, at the very centre of the press and publishing district. In 1960 Helen and Kurt Wolff parted company with Pantheon Books, which became part of Random House, and returned to Europe. During a visit to Germany in 1963, on his way to the German Literature Archive in Marbach, Wolff was hit by a truck and succumbed to his injuries shortly afterwards. He was buried in Marbach. Today, Pantheon is part of the Knopf Doubleday Publishing Group.