Thanks to a question on Quora, I’ve had the chance to explore the skewness of samples from symmetric distributions, prior to and after odd exponent transformations such as . While the answer is posted there, I’d like to explore related odd transformations here and their effect on the skewness. A simple experiment below reveals the impact of non-trivial means on the skewness for data under a cube transformation.

Cube Transformations for Non-Trivial Means

x1 <- rnorm(10^4, 100, 10)
y1 <- x1^3
hist(x1, breaks = 200, col = "light blue", main = "x1; Mean = 100, s.d. = 10")
hist(y1, breaks = 200, col = "light green", main = "y1 = x1^3")

Right skewness visible when samples with significant positive means are transformed.

From here it is apparent that positive values lead to ever-increasing skewness in the transformed data. Significant negative means produce the following result:

Left skewness visible when samples with significant negative means are transformed

Odd-Power Transformations

Next, we explore how the skewness varies with increased exponents, for similar samples.

# Exploring skewness in symmetric distributions
# For mappings of random variablex -> x^3
# Non-trivial negative values of mean

library(e1071)
data <- data.frame()

for (mu in seq(0, 10^3, 1)){
 x <- rnorm(10^4, mu, 10)
 y <- x^3
 y1 <- x^5
 y2 <- x^7
 y3 <- x^9
 data <- rbind(data, c(mu, skewness(x), skewness(y),
 skewness(y1), skewness(y2), skewness(y3)))

}

colnames(data) <- c("mu", "skew_x", "skew_x3", "skew_x5", "skew_x7", "skew_x9") # Plotting x -> x^5
plot(skew_x ~ skew_x3, data = data,
 main = "Skewness(x) vs. Skewness(y = x^3)",
 sub = "Nontrivial positive values of mean",
 col = "dark blue",
 pch = "*",
 xlim = c(-1, 40)
 )
abline(h = 0, v = 0, col= "red")

# Plotting x -> x^5
plot(skew_x ~ skew_x5, data = data,
 main = "Skewness(x) vs. Skewness(y = x^5)",
 sub = "Nontrivial positive values of mean",
 col = "dark green",
 pch = "*",
 xlim = c(-1, 40)
)
abline(h = 0, v = 0, col= "red")

# Plotting x -> x^7
plot(skew_x ~ skew_x7, data = data,
 main = "Skewness(x) vs. Skewness(y = x^7)",
 sub = "Nontrivial positive values of mean",
 col = "dark red",
 pch = "*",
 xlim = c(-1, 40)
)
abline(h = 0, v = 0, col= "red")

#Plotting x -> x^9
plot(skew_x ~ skew_x9, data = data,
 main = "Skewness(x) vs. Skewness(y = x^9)",
 sub = "Nontrivial positive values of mean",
 col = "purple",
 pch = "*",
 xlim = c(-1, 40)
)
abline(h = 0, v = 0, col= "red")

Chances of higher skewness increases with higher non-trivial positive values of mean

These patterns are what are called fat tailed distributions, which are common in complex systems. Given that skewness is defined as the third standard moment, , it is understandable that this behaviour exists when we have higher values of . However, I wonder if this is the exact reason, or if there are deeper technical and statistical reasons behind this pattern. For one thing, you’d expect that x increases proportionally with as for symmetric distributions. Further, do non-trivial values of affect the occurrence of such fat tails? If you happen to know, please comment, I’d love to know more.

Early in December 2016, I spoke at the Strata+Hadoop World 2016 Singapore conference on sensor data analysis approaches, specifically, time series analysis. My company, The Data Team, were represented at Strata+Hadoop World at the innovator’s pavilion. It was a wonderful learning experience for me at the conference, and I have the following key take-aways:

1. There is a lot of interest in advanced machine learning algorithms, deep learning and the capabilities it offers
2. Platform-level innovation is still driving a large part of the big data and data science world forward. Hadoop ecosystem projects are aplenty, and have plenty of variety at the moment
3. There is significant interest in Apache Spark and the capabilities it provides to data science teams. With growing data processing needs and the need to distribute data processing, the Databricks team have improved the Apache Spark interface and performance, so that it is easier to use than before, and scales with fewer teething problems
4. Finally, there is a growing interest in the Internet of Things. Both on the platforms side of things, where new frameworks, architectures and ideas around such systems are discussed, and on the data science side of things, where sensor data analysis approaches and best practices are being discussed, there is increasing momentum.

My talk at Strata+Hadoop World 2016 was on the subject of sensor data analysis. The talk discussed a broad range of approaches, for the analysis of aggregate (I.I.D) data and time series data. You can find a video of the talk at this link. Slides from my talk are here (as a ZIP file).

The Year 2016 in my mind will be associated with three key things, with respect to data:

1. A career transition in engineering, product development and quality management to a career in data science, big data analytics and strategy consulting
2. Learning how to learn better – learning to update my own skills by constant study, reinforcement of key ideas and application
3. Gaining new focus areas, and the journey from explorer to purveyor, and the journey from surveyor to practitioner

I’ll discuss each of these aspects of the year in data below.

Each of these three changes have had a profound impact on my life and career in data so far. Thanks to the excellent team I work with, I’ve been able to appreciate ideas from disparate fields, and I’ve also been able to contribute to their understanding of the domains I have experience in. In this sense, there is a sense of satisfaction. I’ve written earlier on Medium about the importance of good career transitions and how this blog, amongst other things, helped me develop the skills required for my future career.

There is a constant tension that anyone who is learning new skills learns to embrace. One contributor to this tension is the relief and contentment that you understand something. Another contributor to this tension is to deliberately learn to deny the patterns you’re used to seeing in situations, and see these situations with new patterns based on your new knowledge. As the framework or idea becomes more complex, the harder this second kind of learning and application is, to execute. The former is easier than the latter, and in my mind, experienced practitioners have difficulty changing their mindset, and have challenges when adopting new ways of thinking to displace their old ways of thinking.

Whether this is learning new technologies and frameworks, or developing skills about entirely new sciences – such as software systems development or databases , both areas of knowledge relatively new to me at the start of the year – I found myself galloping to catch up, often having to learn new models of these ideas and do away with simplistic models of these ideas. This was both a fascinating and at times debilitating experience as I have explained above. One aspect that enabled this better was better time management, and another, which isn’t often discussed, is the reinforcement of the ideas of importance, by repetition and reinforcement. Such repetition and reinforcement can enable us to learn complex subjects and apply ideas from them, while managing disparate objectives.

Gaining new focus areas was another key feature of 2016. Novelty brings with it the need to step out of comfort zones that are old and long established. It also brings risk, and the possibility of failure. This journey will continue, of course, and there are likely to be lots of stepping stones along the way to insight. Just as we plod through data and analyse, dissect and refactor data sets in many ways to gain new insights from models, there are multiple approaches to address new focus areas – and novelty enables us to examine our inventory of approaches in light of such ideas, and experiment with them. The transition from explorer to purveyor encourages us to take on what Nassim Nicholas Taleb calls “skin in the game”. It makes someone else’s problem our problem – as is so often the case in consulting – and helps build empathy for other people, their situations and organisations. Similarly, the journey from surveyor to practitioner involves applying your lessons from a few exercises (which were hopefully well fabricated, and which hopefully developed true skill) to the real world. This is the bridge between theory and practice, between the system and the landscape, between the rubber and the road, to quote two more analogies.