mUI stands for milli unit interval where a unit interval (UI) is one clock period.
Here is an example on how to use the formula:
If you have a 100MHz clk, that equals a UI of 10ns.
If according to the datasheet, the jitter spec is 60mUI, which equates to:
60 x 10^-3 x 10ns = 600ps of jitter