I just review the NESC 410 tables and it seems there is a significant difference compared to the NFPA 70E tables? It seems they should be similar regardless of the method used. Will the upcoming code updates address the significant difference? Are there any white papers descibing how the 2 different tables were developed and the pros and cons?

Apples and oranges.
Different rules apply to different conditions.

Effectively, NESC tables are for 'open air' work (e.g. the distance between arcing surfaces is large) and 70E is for all other locations, including inside of 'switch houses'.

From NESC:
011. Scope
A. These rules cover..... utility in the exercise of its function as a utility.
D. NESC rules do not cover .... utilization wiring except as covered in Parts 1 and 3.

Basically, when a person employed by a utility is doing tasks governed by NFPA70E, they should be using the PPE from 70E.

What about pad mount transformers? Those are used both by utilities and industry. Same equipment, but very different PPE requirements depending on whether the NESC or NFPA70E tables are used. Why the difference?

Two reasons. First, you had a totally different committee looking at it and they used their experiences which would be very different. I've seen a footnoted version of NESC 2012 and it contains tons of information on how and why the values are what they are. Not so with 70E.

Second, as I understand it the starting calculation information for NESC tables is ArcPro. There is some discrepancy whether the values in the NFPA 70E table come from IEEE 1584 or not. I've seen at least one ROP in the last revision cycle that definitely used IEEE 1584. These two calculation methods will give different results.

For similar reasons NESC uses (or at least used to use) a 2 cal/cm^2 cutoff for PPE requirements whereas NFPA 70E uses 1.2 cal/cm^2. Both based it on available technical information as well as field experience.

Either way, on generation/transmission/distribution equipment, you MUST use NESC as NFPA 70E makes this type of equipment out of scope. For instance energizing by attaching jumpers hot is not allowed under NFPA 70E rules (must use a "disconnect") but is acceptable under NESC and is common practice in many areas. Under 70E, you have a choice. You can use the tables in 70E OR you can use the tables in NESC as they are referenced from 70E in Annex D. However if you are going to use those tables from NESC be sure that they are truly applicable to your situation. I have distribution gear in my "plant". We have our own 50 MW generator, we have overhead distribution systems, we have numerous outdoor as well as indoor substations, and lots of equipment operating well over 1 kV which is where the NESC rules are frequently more appropriate.