In SharePoint 2010 is available the object SP.UI.ModalDialog.You’ll have so the modal dialog windows of SharePoint 2010 available for your application page. That’s cool.
To open the modal dialog you execute the JavaScript code below (you can even add some callback event)
var dialogOptions = SP.UI.$create_DialogOptions();
dialogOptions.url = strPageURL; // URL of the Page
dialogOptions.width = w; // Width of the Dialog
dialogOptions.height = h; // Height of the Dialog
dialogOptions.dialogReturnValueCallback = Function.createDelegate(null, delegate); // Function to capture dialog closed event
SP.UI.ModalDialog.showModalDialog(dialogOptions); // Open the Dialog
return false;
}
To do that you have available the Dialog.master master page. It’s a normal master page with specific content place holder.
The problem I had is that if you have for example a flow of pages in the modal dialog, after a redirect, sometime the HEADER of the modal dialog just disappears. And it does that not everywhere, but just with the Team Site. I don’t know really why, I think it’s something closer to a bug that a feature 
. By the way you can easily solve the problem. Avoid the Dialog.master and, instead, you use a normal ApplicationPage, with the Default.master and you execute the url with the parameter IsDlg=1 in the query string. You’ll have the same behavior of the modal dialog, but we won’t have the disappeared header. Below you have an image of the problem.
In the first image the header is fine, but then, if you change page you just lose it…
And here the problem
