The first reliable record of documentary sources dates from 1,371, the year of creation of the Santisteban manor by Enrique II in favor of Men Rodríguez Biedma, later surnamed Benavides. The manor comprised the Villa de Santisteban del Puerto, together with the places of Castellar and Las Navas. Later in 1473, Enrique IV granted Sánchez de Benavides and Dávila the title of count with the same name. It is at this time when the construction of the walled enclosure that is still preserved begins.

After the Christian Reconquest, the Castle loses its military function. The Dukes of Medinaceli, the last feudal lords of the place, preserving in its interior the keep from the 14th century, called Pallarés, and the parade ground of the old fortress, built a palace of good stalls. In line with Castilian stylistic fashion, the cover features a robust semicircular arch and coats of arms on the salient keystone and spandrels, framed by stylized columns and flames.

Its name as Palacio de Medinaceli is due to the second half of the 18th century, when Luis María, Duke of Medinaceli, son of Pedro Alcántara and María Javiera de Gonzaga, obtained the title of Duke of Santisteban by his marriage to Joaquina de Benavides, daughter by Antonio Benavides and Ana María, Pedro Alcántara's sister.

At the end of the last century the Palace became the Liberal Casino, after the civil war it was the headquarters of the Círculo de Labradores and is currently the Town Hall of Castellar, restored following the original guidelines.

As for the tribute tower, called Pallarés, with a square plan, it consists of two floors ten meters wide and fifteen meters high. Its first floor includes its beautiful pointed arch, and its second floor is worthy of admiration for its dome on pendentives as well as its insurmountable loopholes or loopholes. In 1996 it was registered in the General Catalog of Andalusian Historical Heritage and since 1998 it has been housing the Iberian Museum of Castellar, where you can see some of the important archaeological pieces found in the town, mostly bronze votive offerings, as well as arrowheads, axes or ceramic containers.

The dual personality of the complex is obvious, Castle and Palace, Palace and Castle, appearing superimposed on one and the other of the concepts refined and rough, civil and military, rustic and urban, naive and sophisticated, both competing for the prominence of this singular conglomerate.