Solution 
You can define the exact shadow for each elevation value:
Elevation 
Shadow 
0 
N/A

1 
Color = #00000024, X = 0, Y = 1, Blur = 1, Spread = 0
Color = #0000001F, X = 0, Y = 2, Blur = 1, Spread = 1
Color = #00000033, X = 0, Y = 1, Blur = 3, Spread = 0

2 
Color = #00000024, X = 0, Y = 2, Blur = 2, Spread = 0
Color = #0000001F, X = 0, Y = 3, Blur = 1, Spread = 2
Color = #00000033, X = 0, Y = 1, Blur = 5, Spread = 0

3 
Color = #00000024, X = 0, Y = 3, Blur = 4, Spread = 0
Color = #0000001F, X = 0, Y = 3, Blur = 3, Spread = 2
Color = #00000033, X = 0, Y = 1, Blur = 8, Spread = 0

4 
Color = #00000024, X = 0, Y = 4, Blur = 5, Spread = 0
Color = #0000001F, X = 0, Y = 1, Blur = 10, Spread = 0
Color = #00000033, X = 0, Y = 2, Blur = 4, Spread = 1

6 
Color = #00000024, X = 0, Y = 6, Blur = 10, Spread = 0
Color = #0000001F, X = 0, Y = 1, Blur = 18, Spread = 0
Color = #00000033, X = 0, Y = 3, Blur = 5, Spread = 1

8 
Color = #00000024, X = 0, Y = 8, Blur = 10, Spread = 1
Color = #0000001F, X = 0, Y = 3, Blur = 14, Spread = 2
Color = #00000033, X = 0, Y = 5, Blur = 5, Spread = 3

9 
Color = #00000024, X = 0, Y = 9, Blur = 12, Spread = 1
Color = #0000001F, X = 0, Y = 3, Blur = 16, Spread = 2
Color = #00000033, X = 0, Y = 5, Blur = 6, Spread = 3

12 
Color = #00000024, X = 0, Y = 12, Blur = 17, Spread = 2
Color = #0000001F, X = 0, Y = 5, Blur = 22, Spread = 4
Color = #00000033, X = 0, Y = 7, Blur = 8, Spread = 4

16 
Color = #00000024, X = 0, Y = 16, Blur = 24, Spread = 2
Color = #0000001F, X = 0, Y = 6, Blur = 30, Spread = 5
Color = #00000033, X = 0, Y = 8, Blur = 10, Spread = 5

24 
Color = #00000024, X = 0, Y = 24, Blur = 38, Spread = 3
Color = #0000001F, X = 0, Y = 9, Blur = 46, Spread = 8
Color = #00000033, X = 0, Y = 11, Blur = 15, Spread = 7

