Kasteli is a strong Byzantine fortress, built on the rocky hill of the Aspropountari cape. Its layout is elongated and irregular in shape, with a maximum length of 143 meters and a width of 95 meters.

Its location is naturally fortified and allows control of the sea area between Kalymnos and the nearby island of Telendos, where the fortified settlement of Saint Constantine, contemporary with Kasteli, is located directly opposite. Kasteli also has visual contact with the settlement on the Galatiani mountain range. The hilltop area is naturally divided into two main levels, which were fortified accordingly.

The lower level is surrounded by a strong wall, with rectangular defense towers. The walls are topped with rectangular battlements, while a walkway is formed on their inner side for the movement of the defenders of the fortress. On the slopes on both sides of the hill, a wall extends down to the sea. In other words, the fortress was built in such a way that it could defend against attacks from both land and sea. Gates open at selected points of the wall. Two large water reservoirs are located on the eastern side of the fortification, next to the gates there. Defense towers were formed above the reservoirs.

The upper level, the acropolis, is protected by steep cliffs around it and has only one built staircase for access, which leads to a gate. This area has been reinforced with a wall and rectangular towers. It thus creates a second defensive space in case the enemies breached the outer wall. Its layout is irregular in shape, with a maximum length of 60 meters and a width of 35 meters. To provide the necessary drinking water, a large vaulted water reservoir, measuring 6 x 4 meters internally, was constructed in the eastern part of the acropolis.

Kasteli was created out of the vital need for survival of the remote island of Kalymnos, as well as the Byzantine State itself, due to the naval raids of the Arabs during the mid-7th century CE. It is likely that Kasteli was inhabited until the 10th century, as were the fortified settlements of Saint Constantine on Telendos and Galatiani in Arginonta.

On the southern edges of the hill, near the sea and outside the walls, there is a small vaulted chapel dedicated to the Virgin Mary.