Why you shouldn’t share your goals


Aytekin Tank in The Startup

Originally published on JOTFORM.COM

The race to get the world’s first plane in the sky was a hard fought battle between The Wright Brothers and a lesser-known gentleman by the name of Samuel Pierpont Langley.

You will discover why you’ve never heard of the latter here shortly.

As you probably read somewhere inside that history textbook you were forced to lug around through elementary — The Wright Brothers were responsible for creating the first successful airplane. You remember how the story goes

“… it was a cold windy day on December 17th, 1903 in the Kill Devil Hills of North Carolina… Orville watched nervously as his brother Wilbur climbed inside the plane they had spent years perfecting… miraculously it flew for 59 seconds for a distance of 852 feet…”

While today “The Wright Brothers” is the first name that comes to anyone’s mind when they hear the word fly, once upon a time the pair were major underdogs.

In fact, during the race to the sky, most of America had its money on the man I mentioned earlier, Langley.

He was an extremely outspoken astronomer, physicist and aviation pioneer who was on a mission to make history. Langley’s high stature as the Secretary of the Smithsonian Institution gave him both the credibility and hype he needed to get America on his side.

Not to mention, he was extremely well-backed by the War Department who contributed $50,000 to help him be the first to get a bird in the sky.

Long story short, despite all the hype, Langley’s flying machine ended up crashing and burning while The Wright Brother’s plane ended up soaring.

One party had the entire world, vast resources and plenty of moolah on his side, while the other just had a small bike shop and a passion to fly.

So, let me ask you this… can you guess why The Wright Brothers achieved their goal to take flight while Langley failed?

Early praise feels like you’ve already won.

The Wright Brothers victory over Langley came down to passion, intrinsic motivation (Langley was very status driven) and perhaps praise.

While Langley was sharing his ambitions with the world and being heavily praised for feats he had not yet achieved, The Wright Brothers were receiving little to no attention whatsoever.

Some experts argue that early praise can leave the individual receiving the praise feeling like he or she has already won… in turn causing them to be less likely to follow through with their goals.

For example, in Peter Gollwitzer’s research article, When Intentions Go Public, he raises this very question:

Are scientists more likely to write papers if they tell colleagues about their intentions or if they keep their intentions to themselves?

Gollwitzer and his team of researchers carried out a handful of studies, here is a brief excerpt from their findings:

“Other people’s taking notice of one’s identity-relevant intentions apparently engenders a premature sense of completeness regarding the identity goal.”

In English, what Gollwitzer found was that when individuals set a goal that is closely tied to their identity and then share their intentions with others, they are less likely to achieve the goal.

For example, if your goal is to start drinking more water and you tell your friends and family that you’re going to start drinking more water, this would probably have little to no impact on whether or not you actually drink more water.

Why? Because drinking more water isn’t something you hold close to your identity.

On the other hand, if your goal is to lose 40 lbs and drop 2–3 waist sizes, it might not be the best idea to post about it all over Facebook. Your appearance is something you very much so identify with. So, if you tell people you plan to lose weight and everyone tells you how awesome you are and how great you’re going to look, you might be less likely to lose the weight.

This finding is a bit counterintuitive, considering we were told by our teachers and coaches growing up to set our goals, share our goals, hold ourselves accountable.

But, the theory certainly holds some weight (pun very much intended), and is one that has been adopted by highly successful serial entrepreneurs like Derek Sivers, founder of CD Baby.

Sivers gave a TED Talk on this very topic nearly a decade back. To prove his point, he asked the audience to imagine how they felt when they shared their goals with others:

“Imagine their congratulations and their high image of you. Doesn’t it feel good to say it out loud? Don’t you feel one step closer already? Like, it’s already becoming part of your identity?

Well, bad news. You should have kept your mouth shut. That good feeling makes you less likely to do it.”

Sivers goes on to explain that it’s this “warm feeling” that keeps us from battling on to actually achieve our goals.

When we openly share our goals, we experience a feeling of success that normally only takes place upon completion of the goal.

The result? We don’t ever actually pursue the goal.

Alternatives to sharing your goals.

I’ve recently shared 3 real-life business tactics to achieve your “big hairy goals”. But now, let’s talk about what can actually work when it comes to successfully reaching your goals.

For two counterintuitive yet effective approaches to this, we look to a philosophy called “fear-setting” and making an effort to surround yourself with competition.

Embrace fear-setting over goal-sharing.

Entrepreneur, angel investor and writer, Tim Ferriss, gave an incredible TED Talk where he discussed how fear-setting is instrumental in achieving one’s goals.

He recommends that instead of obsessively sharing your goals, you should come to terms with all the fears that are preventing you from achieving them.

For example, let’s say your goal is to start your own business. Ferriss recommends that you write down all of your fears that are associated with starting a business.

These might include… “Losing all my money”… “Getting fired from my day job”… “Getting laughed at or judged if I fail”.

Once you write down these fears, you should then write down how you would go about preventing these fears (or mitigating the likelihood) of them actually happening.

For example, for the first fear “losing all my money”, your prevention might be… “I’m only going to invest $2,500 that way I can’t lose it all.”

Finally, after you have written down your preventions, you should then write down how you will repair what you fear from happening… if it actually ends up happening.

So, to repair losing the $2,500, you might write down, “Get a part time job as a bartender in addition to my day job until I make the $2,500 back.”

By concentrating on fear-setting over goal-sharing, it allows you to remove the fear that is keeping you from actually achieving your goals.

Surround yourself with competition.

In addition to fear-setting, it might also be a good idea to surround yourself with competition.

A healthy dose of competition can be good for your business, too. At JotForm, we love to use competition to our advantage with events like hackweeks to achieve our product release goals.

A study published two years ago in the journal Preventive Medicine Reports, sheds some light on the impact that competition has on our goals.

The study put 800 undergraduate and graduate students at the University of Pennsylvania through an 11-week exercise program where each person was assigned to work out alone or in a team.

In addition, the teams were designed to be either supportive or competitive.

By the end of the study, researchers found that students involved in the competitive team programs were 90% more likely to attend their scheduled exercise sessions than any other group.

Not only is this number staggering, but it also proves that competition can create a higher level of commitment among people chasing down goals.

When you surround yourself with competition, it doesn’t mean that you have to share your goals with the competition. You don’t have to tell the other folks in the spin class, cross-fit training or pick-up basketball leagues that your goal is to lose 50 lbs.

But, by simply showing up and placing yourself in a competitive environment, you will be more likely to push harder and show up more often — two factors that can help your reach your goals.

The science behind achieving goals has always been an interesting topic.

While some entrepreneurs advocate the idea that you should never have a goal, I’ve recently explained why setting big goals can make you miserable.

Whether you decide to share your goals or not, what I’ve found out across 12 years of entrepreneurship is that you should craft your own path.

What works for others won’t always work for you. And what works for you today won’t always work tomorrow.

You Are Not Equal. I’m Sorry.


(A post is making rounds on social media, in response to the Women’s March on Saturday, January 21, 2017. It starts with “I am not a “disgrace to women” because I don’t support the women’s march. I do not feel I am a “second class citizen” because I am a woman….”

This is my response to that post.)

Say Thank You

Written by

Dina Leygerman

Say thank you. Say thank you to the women who gave you a voice. Say thank you to the women who were arrested and imprisoned and beaten and gassed for you to have a voice. Say thank you to the women who refused to back down, to the women who fought tirelessly to give you a voice. Say thank you to the women who put their lives on hold, who –lucky for you — did not have “better things to do” than to march and protest and rally for your voice. So you don’t feel like a “second class citizen.” So you get to feel “equal.”

Thank Susan B. Anthony and Alice Paul for your right to vote.

Thank Elizabeth Stanton for your right to work.

Thank Maud Wood Park for your prenatal care and your identity outside of your husband.

Thank Rose Schneiderman for your humane working conditions.

Thank Eleanor Roosevelt and Molly Dewson for your ability to work in politics and affect policy.

Thank Margaret Sanger for your legal birth control.

Thank Carol Downer for your reproductive healthcare rights.

Thank Margaret Fuller for your equal education.

Thank Ruth Bader Ginsburg, Shannon Turner, Gloria Steinem, Zelda Kingoff Nordlinger, Rosa Parks, Angela Davis, Malika Saada Saar, Wagatwe Wanjuki, Ida B. Wells, Malala Yousafzai. Thank your mother, your grandmother, your great-grandmother who did not have half of the rights you have now.

You can make your own choices, speak and be heard, vote, work, control your body, defend yourself, defend your family, because of the women who marched. You did nothing to earn those rights. You were born into those rights. You did nothing, but you reap the benefits of women, strong women, women who fought misogyny and pushed through patriarchy and fought for you. And you sit on your pedestal, a pedestal you are fortunate enough to have, and type. A keyboard warrior. A fighter for complacency. An acceptor of what you were given. A denier of facts. Wrapped up in your delusion of equality.

You are not equal. Even if you feel like you are. You still make less than a man for doing the same work. You make less as a CEO, as an athlete, as an actress, as a doctor. You make less in government, in the tech industry, in healthcare.

You still don’t have full rights over your own body. Men are still debating over your uterus. Over your prenatal care. Over your choices.

You still have to pay taxes for your basic sanitary needs.

You still have to carry mace when walking alone at night. You still have to prove to the court why you were drunk on the night you were raped. You still have to justify your behavior when a man forces himself on you.

You still don’t have paid (or even unpaid) maternity leave. You still have to go back to work while your body is broken. While you silently suffer from postpartum depression.

You still have to fight to breastfeed in public. You still have to prove to other women it’s your right to do so. You still offend others with your breasts.

You are still objectified. You are still catcalled. You are still sexualized. You are still told you’re too skinny or you’re too fat. You’re still told you’re too old or too young. You’re applauded when you “age gracefully.” You’re still told men age “better.” You’re still told to dress like a lady. You are still judged on your outfit instead of what’s in your head. What brand bag you have still matters more than your college degree.

You are still being abused by your husband, by your boyfriend. You’re still being murdered by your partners. Being beaten by your soulmate.

You are still worse off if you are a woman of color, a gay woman, a transgender woman. You are still harassed, belittled, dehumanized.

Your daughters are still told they are beautiful before they are told they are smart. Your daughters are still told to behave even though “boys will be boys.” Your daughters are still told boys pull hair or pinch them because they like them.

You are not equal. Your daughters are not equal. You are still systematically oppressed.

Estonia allows parents to take up to three years of leave, fully paid for the first 435 days. United States has no policy requiring maternity leave.

Singapore’s women feel safe walking alone at night. American women do not.

New Zealand’s women have the smallest gender gap in wages, at 5.6%. United States’ pay gap is 20%.

Iceland has the highest number of women CEOs, at 44%. United States is at 4.0%.

The United States ranks at 45 for women’s equality. Behind Rwanda, Cuba, Philippines, Jamaica.

But I get it. You don’t want to admit it. You don’t want to be a victim. You think feminism is a dirty word. You think it’s not classy to fight for equality. You hate the word pussy. Unless of course you use it to call a man who isn’t up to your standard of manhood. You know the type of man that “allows” “his” woman to do whatever she damn well pleases. I get it. You believe feminists are emotional, irrational, unreasonable. Why aren’t women just satisfied with their lives, right? You get what you get and you don’t get upset, right?

I get it. You want to feel empowered. You don’t want to believe you’re oppressed. Because that would mean you are indeed a “second-class citizen.” You don’t want to feel like one. I get it. But don’t worry. I will walk for you. I will walk for your daughter. And your daughter’s daughter. And maybe you will still believe the world did not change. You will believe you’ve always had the rights you have today. And that’s okay. Because women who actually care and support other women don’t care what you think about them. They care about their future and the future of the women who come after them.

Open your eyes. Open them wide. Because I’m here to tell you, along with millions of other women that you are not equal. Our equality is an illusion. A feel-good sleight of hand. A trick of the mind. I’m sorry to tell you, but you are not equal. And neither are your daughters.

But don’t worry. We will walk for you. We will fight for you. We will stand up for you. And one day you will actually be equal, instead of just feeling like you are.

~ Dina Leygerman, 2017

Follow me on Facebook & Twitter

RIP Crypto


Altcoins are dying a coin and code death faster than ever before.

The cryptocurrency market is sadly tethered to Bitcoin’s price. Bitcoin’s price is probably at the mercy of large-scale market manipulation and cryptic factors that even the crypto frauds and “decentralization” pretenders can’t simulate.

However, the RIP Crypto title here is still just a meme. There’s no end for crypto in sight, with the levels of blockchain projects we are seeing, with the levels of daily active developers being attracted to the new space and bigger investors, crypto mania is only a faltering global economy away. Most economics peg the next global recession around twelve to eighteen months away, so late 2019 or early 2020.

RIP “crypto”. You had a good run.

With Wall Street and Bloomberg reports tempted to take this literally, and try to convince you that crypto really is dead, there was one funny thing that did occur recently.

This week veteran cryptographer Matt Blaze, according to TechCrunch, did finally sell the pithy domain name he registered in 1993, in the midst of the PC era crypto wars — to none other than Crypto Valley.

The buyer/recipient is Monaco, the Zug, Switzerland-based payments and cryptocurrency platform startup whose self-styled mission is “accelerating the world’s transition to cryptocurrency”, positioning itself at the nexus of the current crypto craze.

So crypto.com now points to cryptocurrencies.

The Frontlines of Controversy

You got a lot of sneaky startups Crypto! You are vilified but sort of immortal in our psyches. Everyone likes a dark horse though, as young consumers are themselves penalized in a global economy stacked against them. Crypto, is like a symbol of an uprising. An uprising in coin, code, node and word-of-mouth Bitcoin gibberish.

It’s that recognition on a global level, that decentralization really is the ethical corporate future of all services, government and digital infrastructure on the blockchain.

Listening to Vitalik Buterin is like listening to the voices of the spirit of crypto, itself. You want to support what he does, but you don’t know wtf he is talking about. I suspect EOS will fail and NEO will vastly improve its offering before 2020. But what do I know, I’m just a beginner blockchain historian of sorts. Ethereum’s sharding, plasma and Casper could make contenders like Waves, Tezos, Stratis, Qtum and others obsolete.

If you think Crypto is dead, just go on Reddit and check out all of the subreddits around them. They are some of the most active and fun subreddits to follow on all of Reddit, that itself has massively engaged users. Slack and Telegram users migrating to Discord, is another trend that shows blockchain startups are building amazing communities. Even as Facebook wants to enable Facebook Groups to be monetized like exclusive clubs. This from a platform that has banned Crypto Ads, and perma-banned ICO Ads.

It’s very difficult to see Bitcoin dying any time soon, blockchain patents are being activated now. The Ethereum Foundation tries very hard to be a decentralization community; basically a new kind of model from the Microsoft Cloud developers model (that recently bought out GitHub). Developers want to be free and contribute their talents for some kind of a planetary good. Ethereum’s ecosystem is a place for collaboration; not some kind of centralized model like Microsoft; or even the likes of Ripple, Block.One and other experiments.

Cardano, DFINITY, VeChain, TRON and so many other dApp and MainNets deserve close watching. Blockchain startups aren’t even going at full speed yet, altcoins could explode if another crypto singularity took place (sort of like the bump of December, 2017). Over 99% of the use cases of blockchain have yet to even mature. RIP Crypto, yes O’ really?

The future is usability and scalability, privacy and security are on-going challenges. The majority of the most valuable cryptocurrencies are dApp, smart contract, platforms and developer ecosystems that could scale with solid real-world capabilities for a future we cannot even quite imagine.

The sale of crypto.com domain for like $10 million doesn’t help us understand the price volatility or regulatory immaturity of the space; all it tells us is that crypto is a beast that is two-faced. When the future of coin and code is at stake, there’s bound to be a lot of controversy.

Crypto cannot die because it’s also short for cryptography; and we all know the future of cybersecurity is one of the major issues of the evolution of the web, the internet of things and all things mobile, markets, data, privacy, and fraud. How anonymity meets user-ID verification meets taxes, meets dark-net transactions and side hustle investments, is bound to have some growing pains. If anything points to crypto not dying, it’s Coinbase custody.

21 Quotes That (If Applied) Change You Into a Better Person



As long as man has been alive, he has been collecting little sayings about how to live. We find them carved in the rock of the Temple of Apollo and etched as graffiti on the walls of Pompeii. They appear in the plays of Shakespeare, the commonplace book of H. P. Lovecraft, the collected proverbs of Erasmus, and the ceiling beams of Montaigne’s study. Today, they’re recorded on iPhones and in Evernote.

But whatever generation is doing it, whether they’re written by scribes in China or commoners in some European dungeon or simply passed along by a kindly grandfather, these little epigrams of life advice have taught essential lessons. How to respond to adversity. How to think about money. How to meditate on our mortality. How to have courage.

And they pack all this in in so few words. “What is an epigram?” Coleridge asked, “A dwarfish whole; Its body brevity, and wit its soul.” Epigrams are what Churchill was doing when he said: “To improve is to change, so to be perfect is to have changed often.” Or Balzac: “All happiness depends on courage and work.” Ah yes, epigrams are often funny too. That’s how we remember them. Napoleon: “Never interrupt an enemy making a mistake.” François de La Rochefoucauld: “We hardly find any persons of good sense save those who agree with us.” Voltaire: “A long dispute means that both parties are wrong.”

Below are some wonderful epigrams that span some 21 centuries and 3 continents. Each one is worth remembering, having queued in your brain for one of life’s crossroads or to drop at the perfect moment in conversation. Each will change and evolve with you as you evolve (Heraclitus: “No man steps in the same river twice”) and yet each will remain strong and unyielding no matter how much you may one day try to wiggle out and away from them.

Fundamentally, each one will teach you how to be a better person. If you let them.

“We must all either wear out or rust out, every one of us. My choice is to wear out.” — Theodore Roosevelt

At the beginning of his life, few would have predicted that Theodore Roosevelt even had a choice in the matter. He was sickly and fragile, doted on by worried parents. Then, a conversation with his father sent him driven, almost maniacally in the other direction. I will make my body,” he said, when told that he would not go far in this world with a brilliant mind in a frail body. What followed was a montage of boxing, hiking, horseback riding, hunting, fishing, swimming, boldly charging enemy fire, and then a grueling work pace as one of the most prolific and admired presidents in American history. Again, this epigram was prophetic for Roosevelt, because at only 54 years old, his body began to wear out. An assassination attempt left a bullet lodged in his body and it hastened his rheumatoid arthritis. On his famous “River of Doubt” expedition he developed a tropical fever and the toxins from an infection in his leg left him nearly dead. Back in America he contracted a severe throat infection and was later diagnosed with inflammatory rheumatism, which temporarily confined him to a wheelchair (saying famously, “All right! I can work that way too!”) and then he died at age 60. But there is not a person on the planet who would say that he had not made a fair trade, that he had not worn his life well and not lived a full one in those 60 years.

“It’s not what happens to you, but how you react to it that matters.” — Epictetus

There is the story of the alcoholic father with two sons. One follows in his father’s footsteps and ends up struggling through life as a drunk, and the other becomes a successful, sober businessman. Each are asked: “Why are you the way you are?” The answer for both is the same: “Well, it’s because my father was an alcoholic.” The same event, the same childhood, two different outcomes. This is true for almost all situations — what happens to us is an objective reality, how we respond is a subjective choice. The Stoics — of which Epictetus was one — would say that we don’t control what happens to us, all we control are our thoughts and reactions to what happens to us. Remember that: You’re defined in this life not by your good luck or your bad luck, but your reaction to those strokes of fortune. Don’t let anyone tell you different.

“The best revenge is not to be like that.” — Marcus Aurelius

There is a proverb about revenge: Before setting out for a journey of revenge, dig two graves. Because revenge is so costly, because the pursuit of it often wears on the one who covets it. Marcus’s advice is easier and truer: How much better it feels to let it go, to leave the wrongdoer to their wrongdoing. And from what we know, Marcus Aurelius lived this advice. When Avidius Cassius, one of his most trusted generals rebelled and declared himself emperor, Marcus did not seek vengeance. Instead, he saw this as an opportunity to teach the Roman people and the Roman Senate about how to deal with civil strife in a compassionate, forgiving way. Indeed, when assassins struck Cassius down, Marcus supposedly wept. This is very different than the idea of “Living well being the best revenge” — it’s not about showing someone up or rubbing your success in their face. It’s that the person who wronged you is not happy, is not enjoying their life. Do not become like them. Reward yourself by being the opposite of them.

“There is good in everything, if only we look for it.” — Laura Ingalls Wilder

Laura Ingalls Wilder, author of the classic series Little House, lived this, facing some of the toughest and unwelcoming elements on the planet: harsh and unyielding soil, Indian territory, Kansas prairies, and the humid backwoods of Florida. Not afraid, not jaded — because she saw it all as an adventure. Everywhere was a chance to do something new, to persevere with cheery pioneer spirit whatever fate befell her and her husband. That isn’t to say she saw the world through delusional rose-colored glasses. Instead, she simply chose to see each situation for what it could be — accompanied by hard work and a little upbeat spirit. Others make the opposite choice. Remember: There is no good or bad without us, there is only perception. There is the event itself and the story we tell ourselves about what it means.

“Character is fate.” — Heraclitus

In the hiring process, most employers look at where someone went to school, what jobs they’ve held in the past. This is because past success can be an indicator of future successes. But is it always? There are plenty of people who were successful because of luck. Maybe they got into Oxford or Harvard because of their parents. And what about a young person who hasn’t had time to build a track record? Are they worthless? Of course not. This is why character is a far better measure of a man or woman. Not just for jobs, but for friendships, relationships, for everything. When you seek to advance your own position in life, character is the best lever — perhaps not in the short term, but certainly over the long term. And the same goes for the people you invite into your life.

“If you see fraud and do not say fraud, you are a fraud.” — Nicholas Nassim Taleb

A man shows up for work at a company where he knows that management is doing something wrong, something unethical. How does he respond? Can he cash his checks in good conscience because he isn’t the one running up the stock price, falsifying reports or lying to his co-workers? No. One cannot, as Budd Schulberg says in one of his novels, deal in filth without becoming the thing he touches. We should look up to a young man at Theranos as an example here. After discovering numerous problems at the health care startup, he was dismissed by his seniors and eventually contacted the authorities. Afterwards, not only was this young man repeatedly threatened, bullied, and attacked by Theranos, but his family had to consider selling their house to pay for the legal bills. His relationship with his grandfather — who sits on the Theranos board — is strained and perhaps irreparable. As Marcus Aurelius reminded himself, and us: “Just that you do the right thing. The rest doesn’t matter.” It’s an important reminder. Doing the right thing isn’t free. Doing the right thing might even cost you everything.

“Every man I meet is my master in some point, and in that I learn of him.” — Ralph Waldo Emerson

Everyone is better than you at something. This is a fact of life. Someone is better than you at making eye contact. Someone is better than you at quantum physics. Someone is better informed than you on geopolitics. Someone is better than you are at speaking kindly to someone they dislike. There are better gift-givers, name-rememberers, weight-lifters, temper-controllers, confidence-carriers, and friendship-makers. There is no one person who is the best at all these things, who doesn’t have room to improve in one or more of them. So if you can find the humility to accept this about yourself, what you will realize is that the world is one giant classroom. Go about your day with an openness and a joy about this fact. Look at every interaction as an opportunity to learn from and of the people you meet. You will be amazed at how quickly you grow, how much better you get.

“This is not your responsibility but it is your problem.” — Cheryl Strayed

It is not your responsibility to fill up a stranger’s gas tank, but when their car dies in front of you, blocking the road, it’s still your problem isn’t it? It is not your responsibility to negotiate peace treaties on behalf of your country, but when war breaks out and you’re drafted to fight in it? Guess whose problem it is? Yours. Life is like this. It has a way of dropping things into our lap — the consequences of an employee’s negligence, a spouse’s momentary lapse of judgement, a freak weather event — that were in no way our fault but by nature of being in our lap, our f*cking problem. So what are you going to do? Complain? Are you going to litigate this in a blogpost or an argument with God? Or are you just going to get to work solving it the best you can? Life is defined by how you answer that question. Cheryl Strayed is right. This thing might not be your responsibility but it is your problem. So accept it, deal with it, kick its ass.

“Waste no more time arguing what a good man should be. Be one.” — Marcus Aurelius

In Rome just as America, in the forum just as on Facebook, there was the temptation to replace action with argument. To philosophize instead of living philosophically. Today, in a society obsessed with content, outrage, and drama, it’s even easier to get lost in the echo chamber of the debate of what’s “better.” We can have endless discussions about what’s right and wrong. What should we do in this hypothetical situation or that one? How can we encourage other people to be better? (We can even debate the meaning of the above line: “What’s a man? What’s the definition of good? Why doesn’t it mention women?”) Of course, this is all a distraction. If you want to try to make the world a slightly better place, there’s a lot you can do. But only one thing guarantees an impact. Step away from the argument. Dig yourself out of the rubble. Stop wasting time with how things should be, would be, could be. Be that thing. (Here’s a cool poster of this quote).

“You are only entitled to the action, never to its fruits.” — Bhagavad Gita

In life, it’s a fact that: You will be unappreciated. You will be sabotaged. You will experience surprising failures. Your expectations will not be met. You will lose. You will fail. How do you carry on then? How do you take pride in yourself and your work? John Wooden’s advice to his players says it: Change the definition of success. “Success is peace of mind, which is a direct result of self satisfaction in knowing you made the effort to do your best to become the best that you are capable of becoming.” “Ambition,” Marcus Aurelius reminded himself, “means tying your well-being to what other people say or do . . . Sanity means tying it to your own actions.” Do your work. Do it well. Then “let go and let God.” That’s all there needs to be. Recognition and rewards — those are just extra.

“Self-sufficiency is the greatest of all wealth.” — Epicurus

A lot has been said of so-called “F*ck You Money.” The idea being that if one can earn enough, become rich and powerful enough, that suddenly no one can touch them and they can do whatever they want. What a mirage this is! How often the target seems to mysteriously move right as we approach it. It calls to mind the observation of David “DHH” Heinemeier Hansson who said that “beyond a specific amount, f*ck-you money can be a state of mind. One that you can acquire well in advance of the corresponding bank account. One that’s founded mostly on a personal confidence that even if most of the material trappings went away, you’d still be happier for standing your ground.” The truth is being your own man, being self-contained, having fewer needs, and better, resilient skills that allow you to thrive in any and all situations. That is real wealth and freedom. That’s what Emerson was talking about in his famous essay on self-reliance and it’s what Epicurus meant too.

“Tell me to what you pay attention and I will tell you who you are.” — Jose Ortega y Gasset

It was one of the great Stoics who said that if you live with a lame man, soon enough you will walk with a limp. My father told me something similar as a kid: “You become like your friends.” It is true not just with social influences but informational ones too: If you are addicted to the chatter of the news, you will soon find yourself worried, resentful, and perpetually outraged. If you consume nothing but escapist entertainment, you will find the real world around you harder and harder to deal with. If all you do is watch the markets and obsess over every fluctuation, your worldview will become defined by money and gains and losses. But if you drink from deep, philosophical wisdom? If you have regularly in your mind role models of restraint, sobriety, courage, and honor? Well, you will start to become these things too. Tell me who you spend time with, Goethe said, and I will tell you who you are. Tell me what you pay attention to, Gasset was saying, and I can tell you the same thing. Remember that the next time you feel your finger itching to pull up your Facebook feed.

“Better to trip with the feet than with the tongue.” — Zeno

You can always get up after you fall, but remember, what has been said can never be unsaid. Especially cruel and hurtful things.

“Space I can recover. Time, never.” — Napoleon Bonaparte

Lands can be reconquered, indeed in the course of a battle, a hill or a certain plain might trade hands several times. But missed opportunities? These can never be regained. Moments in time, in culture? They can never be re-made. One can never go back in time to prepare for what they should have prepared for, no one can ever get back critical seconds that were wasted out of fear or ego. Napoleon was brilliant at trading space for time: Sure, you can make these moves, provided you are giving me the time I need to drill my troops, or move them to where I want them to be. Yet in life, most of us are terrible at this. We trade an hour of our life here or afternoon there like it can be bought back with the few dollars we were paid for it. And it is only much much later, as they are on their deathbeds or when they are looking back on what might have been, that many people realize the awful truth of this quote. Don’t do that. Embrace it now.

“You never know who’s swimming naked until the tide goes out.” — Warren Buffett

The problem with comparing yourself to other people is you really never know anyone else’s situation. The co-worker with a nice car? It could be a dangerous and unsafe salvage with 100,000 miles. The friend who always seems to be traveling to far off places? They could be up to their eyeballs in credit card debt and about to get fired by their boss. Your neighbors’ marriage which makes you so insecure about your own? It could be a nightmare, a complete lie. People do a very good job pretending at things, and their well-maintained fronts are often covers for incredible risk and irresponsibility. You never know, Warren Buffett was saying, until things get bad. If you’re living the life you know to be right, if you are making good, solid decisions, don’t be swayed by what others are doing — whether that is taking the form of irrational exuberance or panicked pessimism. See the high flying lives of others as a cautionary tale — like Icarus with his wings — and not as an inspiration or a source of insecurity. Keep doing what you’re doing and don’t be caught swimming naked! Because the tide will go out. Prepare for it! (Premeditatio Malorum)

“Search others for their virtues, thyself for thy vices.” — Benjamin Franklin

Marcus Aurelius would say something similar: “Be tolerant with others and strict with yourself.” Why? For starters because the only person you control is yourself. It’s a complete waste of time to go around projecting strict standards on other people — ones they never agreed to follow in the first place — and then being aghast or feel wronged when they fall short. The other reason is you have no idea what other people are going or have been through. That person who seemed to rudely decline the invitation you so kindly offered? What if they were working hard to recommit themselves to their family and as much as they’d like to have coffee with you, are doing their best to spend more time with their loved ones? The point is: You have no idea. So give people the benefit of the doubt. Look for good in them, assume good in them, and let that good inspire your own actions.

“The world was not big enough for Alexander the Great, but a coffin was.” — Juvenal

Ah, the way that a good one liner can humble even the world’s greatest conqueror. Remember: we are all equals in death. It makes quick work of all of us, big and small. I carry a coin in my pocket to remember this: Memento Mori. What Juvenal reminds us is the same thing that Shakespeare spoke about in Hamlet:

“Imperious Caesar, dead and turned to clay,
Might stop a hole to keep the wind away.
O’ that that earth which kept the world in awe
Should patch a wall t’ expel the winder’s flaw!”

It doesn’t matter how famous you are, how powerful you are, how much you think you have left to do on this planet, the same thing happens to all of us, and it can happen when we least expect it. And then we will be wormfood and that’s the end of it.

“To improve is to change, so to be perfect is to have changed often.” — Winston Churchill

While this is probably not a Churchill original (he most likely borrowed from Cardinal Newman: “In a higher world it is otherwise, but here below to live is to change, and to be perfect is to have changed often”), Churchill certainly abided this in his life. He’d even quip about his constant change of political affiliation: “I said a lot of stupid things when I worked with the Conservative Party, and I left it because I did not want to go on saying stupid things.” As Cicero would say when attacked that he was changing his opinion: “If something strikes me as probable, I say it; and that is how, unlike everyone else, I remain a free agent.” There is nothing more impressive — intellectually or otherwise — than to change long held beliefs, opinions, and habits. The more you’ve changed, the better you probably are.

“Judge not, lest you be judged.” — Jesus

Not only here would Jesus call us on one of our worst tendencies but immediately also ask: “And why do you look at the speck in your brother’s eye, but do not consider the plank in your own eye?” This line is similar to what the Stoic philosopher Seneca, who historical sources suggest was born the same year as Jesus, would say: “You look at the pimples of others when you yourselves are covered with a mass of sores.” Waste no time judging and worrying about other people. You have plenty of problems to deal with in your own life. Chances are your own flaws are probably worse — and in any case, they are at least in your control. So do something about them.

“Time and patience are the strongest warriors.” — Leo Tolstoy

Tolstoy puts the above words in the mouth of Field Marshall Mikhail Kutuzov in War and Peace. In real life, Kutuzov gave Napoleon a painful lesson in the truth of the epigram over a long winter in Russia in 1812. Tolstoy would also say, “Everything comes in time to him who knows how to wait.” When it comes to accomplishing anything significant, you are required to exhibit patience and fortitude, so much patience, as much as you’d think you’d need boldness and courage. In my book Conspiracy, about Peter Thiel’s plot to destroy Gawker, his operative describes a similar idea: With enough time and patience, you can do anything.

“No one saves us but ourselves / No one can and no one may.” — Buddha

Will we wait for someone to save us, or will we listen to Marcus Aurelius’s empowering call to “get active in your own rescue — if you care for yourself at all — and do it while you can.”

Because at some point, we must put articles like this one aside and take action. No one can blow our nose for us. Another blog post isn’t the answer. The right choices and decisions are. Who knows how much time you have left, or what awaits us tomorrow? So get to it.

Like to Read?

I’ve created a list of 15 books you’ve never heard of that will alter your worldview and help you excel at your career.

Get the secret book list here!


This piece originally ran on Art of Manliness.

If You Only Read A Few Books In 2018, Read These



If you’d liked to be jerked around less, provoked less, and more productive and inwardly focused, where should you start?

To me, the answer is obvious: by turning to wisdom. Below is a list of 21 books that will help lead you to a better, stronger 2018.

Deep Work: Rules for Focused Success in a Distracted World by Cal Newport Media consumption went way up in 2017. For most of us, that meant happiness and productivity went way down. The world is becoming noisier and will become more so every day. If you can’t cultivate the ability to have quiet, insightful, deeply focused periods of productive work, you’re going to get screwed. This is a book that explains how to cultivate and protect that skill — the ability to do deep work. I strongly urge you to begin this practice in 2018— if you want to get anything done or perform your best.

The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life by Mark Manson To me, practical philosophy has always been the art knowing what to — and what not to — give a fuck about. That’s what Mark’s book is about. It’s not about apathy. It’s about cultivating indifference to things that don’t matter. Be careful, as Marcus Aurelius warns, not to give the little things more time and thought they deserved. Maybe looking back at this year reveals how much effort you’ve frittered away worrying about the trivial. If so, let 2018 be a year that you only devote energy to things that truly matter — get the important things right by ignoring the insignificant.

The Way to Love: The Last Meditations of Anthony de Mello by Anthony de Mello Coach Shaka Smart recommended this little book (and it’s a little book, probably the smallest I’ve ever read. It fits in your palm). But it’s an incredibly wise and helpful read. Written by a Catholic Priest who’d lived in India, the book has this unusual convergence of eastern and western thought. One of my favorite lines: “The question to ask is not ‘What’s wrong with this person?’ but ‘What does this irritation tell me about myself?’ I plan on regularly revisiting it throughout 2018.

But What If We’re Wrongby Chuck Klosterman It’s always good to remind ourselves that almost everything we’re certain about will probably be eventually proven wrong. Klosterman’s subtitle — Thinking About the Present As If It Were the Past — is a brilliant exercise for getting some perspective in 2018. Whether you think it’s going to be a year of radical change for the better or a horrible year of excesses of dangerous precedent, you’re probably wrong. You’re probably not even in the ballpark. This book shows you why, not with lectures about politics, but with a bunch of awesome thought experiments about music, books, movies and science.

Rules for Radicals: A Practical Primer for Realistic Radicals by Saul Alinsky If Hillary Clinton had remembered the lessons of Saul Alinsky (who she wrote her college thesis on), the election may have turned out differently. Why? A notorious strategist and community organizer, Alinsky was a die hard pragmatist, but he also knew how to tell a story and create a collective cause. He could work within the system but knew how to shake it up and generate attention. This book is a classic and woefully underrated. Whatever you set out to do in 2018, this book can provide you with strategic guidance and insight.

The Filter Bubble by Eli Pariser / Trust Me I’m Lying by Ryan Holiday / The Brass Check by Upton Sinclair I strongly recommend that you take the time in 2018 to read these books. In light of this year, you owe it to yourself to study and better understand how our media system works. In The Filter Bubble, Eli Pariser warns of the danger of living in bubbles of personalization that reinforce and insulate our worldview. Though Sinclair’s The Brass Check has been almost entirely forgotten by history, it’s not only fascinating but a timeless perspective. Sinclair deeply understood the economic incentives of early 20th century journalism and thus could predict and analyze the manipulative effect it had on The Truth. I used that book as a model for my expose of the media system, Trust Me, I’m Lying. Today, the incentives and pressures are different but they warp our information in a similar way. In almost every substantial charge Upton leveled against the yellow press, you could, today, sub in blogs and the cable news cycle and be even more correct.

48 Laws of Power / 33 Strategies of War by Robert Greene Robert Greene is a master of human psychology and human dynamics — he has a profound ability to explain timeless truths through story and example. You can read the classics and not always understand the lessons. But if you read Robert’s books, I promise you will leave not just with actionable lessons but an indelible sense of what to do in many trying and confusing situations. I wrote earlier this year that strategic wisdom is not something we are born with — but the lessons are there for us to pick up. Pick these two up before the year ends and operate the next with a strategic mindset and clarity.

Conspiracy: Peter Thiel, Hulk Hogan, Gawker, and the Anatomy of Intrigue by Ryan Holiday — If you want to immerse yourself in the above topics of media and strategy, and are looking for one book to teach you lessons in both, my book on the nearly decade-long conspiracy that billionaire Peter Thiel waged against Gawker will do this for you. This is a stunning story about how power works in the modern age, and is a masterclass in strategy and how to accomplish wildly ambitious aims.

The Road To Character by David Brooks When General Stanley McChrystal was asked on the Tim Ferriss podcastwhat was a recent purchase that had most positively impacted his life, he pointed to this book. I agree. It can be a bit stilted and dense at times, but it should be assigned reading to any young person today (a little challenge is a good thing). Illustrating with examples and stories from great men and women, Brooks admonishes the reader to undertake their own journey of character perfection. In my own book, I explore the same topic (humility) from a different angle using similar stories — I’m attacking ego, he’s building up character. Both will be important for the next year.

The Dipby Seth Godin This book is a short 70 pages and it looks like something someone would give as a joke gift, but it’s anything but. Godin talks frankly about quitting and pushing through — and when to do each. Quit when you’ll be mediocre, when the returns aren’t worth the investment, when you no longer think you’ll enjoy the ends. Stick when the dip is the obstacle that creates scarcity, when you’re simply bridging the gap between beginner’s luck and mastery. I promise, next year you are guaranteed to find yourself in moments when you don’t know what is the right answer. This book will help you find it.

Hillbilly Elegy: A Memoir of a Family and Culture in Crisis by J. D. Vance / Strangers in Their Own Land: Anger and Mourning on the American Right by Arlie Russell Hochschild You might describe Hillbilly Elegy as a Ta-Nehisi Coates style memoir about a community that — at least in progressive circles — gets a lot less attention: disaffected, impoverished whites (particularly in the mid-east and South). I thought the book was empathetic, self-aware and inspiring. The author pokes some holes in the concept of “white privilege” — certainly a third or fourth generation hillbilly in Kentucky doesn’t walk around feeling like they have it easy — and an explanation of some of the phenomenon behind Donald Trump (notice I said explanation, not an excuse). This is a sober but also hopeful book. I urge people on all sides of the political spectrum to read it, but progressives would benefit from the eye-opening perspective the most. Pair it with Strangers in Their Own Land, the author of which, a Berkeley native, visits and interviews people in the deep Louisiana countryside. (Remember: you don’t have to agree with the people you’re reading about — but you should care what makes them tick.)

It Can’t Happen Here by Sinclair Lewis This book is one that will make you so uncomfortable that you’ll probably pick it up and put it down several times. It almost shocks you that this exists, that it’s not some work of fiction pretending to be 80 years old. But no, in fact, one of America’s most famous writers wrote a bestselling book about an appalling populist demagogue who won the presidency of the United States. Europe was a mess, the economy was in the toilet. Well meaning people talked earnestly about the need for “serious change” and “revolution.” The parties split, and a fringe candidate suddenly becomes viable. When people tried to question some of the ideas he campaigned on, they were shouted down: “This is how all politicians speak. He’s not serious about that extreme stuff.” Life imitates art and now, this is what actually happened. Change the dates, places and names and it’s no longer fiction, it’s real. Fiction is best when it puts a mirror up to us. This book does that. If you don’t read the book, at least please read about it. Because you need to know. It can happen here.

How to Live: Or A Life of Montaigne in One Question and Twenty Attempts at an Answer by Sarah Bakewell The book is spectacular. It was a bestseller in the UK and was featured in a 6 part series in The Guardian. The format of the book is a bit unusual, instead of chapters it is made up of 20 Montaigne style essays that discuss the man from a variety of different perspectives. Montaigne was a man obsessed with figuring himself out — why he thought the way he did, how he could find happiness, his fetishes, his near-death experiences. He lived in tumultuous times too and he coped by looking inward. We’re lucky he did, and we can do the same.

Tribe: On Homecoming and Belonging by Sebastian Junger Junger looks at many of the social issues we have today: PTSD. Depression. Political polarization. Contempt for our fellow citizens. Aimlessness. We’ve lost the bonds that tie a tribe and culture together. No wonder veterans feel alone and lost when they leave their tight knit tribe and re-enter solitary civilian life. No wonder manypeople feel that way regardless of whether they served or not. No wonder we have trouble agreeing on basic solutions to common problems. There is no sense of what is common or basic or shared. Given the divisiveness that we are facing as a society — that became painfully clear in 2016 — this is one of the most urgent and important book you need to read next year.

The Years of Lyndon Johnson (4 Volumes) by Robert A. Caro In January of 2016, I picked up my first book in this Caro’s series on Lyndon Johnson. It wasn’t until June that I finished my fourth, but I consider finishing all (~3,500 pages) of them to be one of my proudest reading achievements. If there is one line that sums up the whole series it’s this: It’s that power doesn’t only corrupt. That’s too simple. What power does is reveal. It’s also easy to be disillusioned by politics right now but for me, getting lost in these Lyndon Johnson books has been a helpful and educational process. Because you learn two things 1) that things have always been complicated and confusing but they tend to turn out alright 2) that our system, whatever its flaws, can still produce good results from bad men. And without a doubt, that’s a good reminder to have on the eve of 2018.

The Last Lion: Winston Spencer Churchill (3 Volumes) by William Manchester Don’t think an individual can make a difference in history? Then you haven’t read enough about Winston Churchill. The scenes in this book: warning about the looming threat of Nazism, the retreat at Dunkirk, persevering through the Blitz, vowing to fight on the landing grounds and the beaches and in the streets, whatever the cost may be. The sheer determination of this man, to take an entire country on your back and defy a horde which had overrun the European continent in a matter of months…it’s almost breathtaking to think about. (If you were to only read one thing in the next year, you could do a lot worse than either of these series. They contain dozens of books within them and will teach you about so much more than just the man they are ostensibly about. Please, please read them.)

Mr. Eternity by Aaron Thier / The Count of Monte Cristo by Alexandre Dumas These books really have nothing to do with the events of 2016 but they are long and entertaining and they will make you forget your problems for the next 12 months. I thought I’d read Monte Cristo as a kid, but clearly I didn’t. Because the actual book is a 1,200 page epic of some of the most brilliant, beautiful and complicated storytelling ever put to paper. What a book! When I typed out my notes (and quotes) after finishing this book, it ran some 3,000 words. As for Mr. Eternity, it’s a fun, epic jaunt through the distant future (as though it was the past) in the vein of Voltaire’s Candide. Candide isn’t a bad read for this year either: “We must cultivate our own garden.”

Like to Read?

I’ve created a list of 15 books you’ve never heard of that will alter your worldview and help you excel at your career.

Get the secret book list here!

Optimizing BigQuery: Cluster your tables


BigQuery just announced the ability to cluster tables — which I’ll describe here. If you are looking for massive savings in costs and querying times, this post is for you.

Warning: This post plays with several terabytes of data. Set up your BigQuery cost controls, buckle up, keep calm, and query on.

Can you spot the huge difference between the query on the left and the one on the right?

Same query over almost the same tables — one is clustered, the other is not. Faster and more efficient.

What can we see here:

  • Kubernetes was a popular topic during 2017 in Wikipedia — with more than 400,000 views among the English Wikipedia sites.
  • Kubernetes gets way less pageviews on the mobile Wikipedia site than on the traditional desktop one (90k vs 343k).

But what’s really interesting:

  • The query on the left was able to process 2.2 TB in 20 seconds. Impressive, but costly.
  • The query on the right went over a similar table — but it got the results in only 5.4 seconds, and for one tenth of the cost (227 GB).

What’s the secret?

Clustered tables!

Now you can tell BigQuery to store your data “sorted” by certain fields — and when your queries filter over these fields, BigQuery will be smart enough to only open the matching clusters. In other words, you’ll get faster and cheaper results.

Previously I explained my lazy loading of Wikipedia views into partitioned yearly BigQuery tables. As a reminder, just for 2017 we will be querying more than 190 billion of pageviews, over 3,927 Wikipedia sites, covered by over 54 billion rows (2.2 TBs of data per year).

To move my existing data into a newly clustered table I can use some DDL:

CREATE TABLE `fh-bigquery.wikipedia_v3.pageviews_2017`
CLUSTER BY wiki, title
description="Wikipedia pageviews - partitioned by day, clustered by (wiki, title). Contact https://twitter.com/felipehoffa"
, require_partition_filter=true
AS SELECT * FROM `fh-bigquery.wikipedia_v2.pageviews_2017`
WHERE datehour > '1990-01-01' # nag
-- 4724.8s elapsed, 2.20 TB processed

Note these options:

  • CLUSTER BY wiki, title: Whenever people query using the wiki column, BigQuery will optimize these queries. These queries will be optimized even further if the user also filters by title. If the user only filters by title, clustering won’t work, as the order is important (think boxes inside boxes).
  • require_partition_filter=true: This option reminds my users to always add a date filtering clause to their queries. That’s how I remind them that their queries could be cheaper if they only query through a fraction of the year.
  • 4724.8s elapsed, 2.20 TB processed: It took a while to re-materialize and cluster 2.2 TB of data — but the future savings in querying makes up for it. Note that you could run the same operation for free with a bq export/load combination — but I like the flexibility that DDL supports. Within theSELECT * FROM I could have filtered and transformed my data, or added more columns just for clustering purposes.

Better clustered: Let’s test some queries


SELECT * LIMIT 1 is a known BigQuery anti-pattern. You get charged for a full table scan, while you could have received the same results for free with a table preview. Let’s try it on our v2 table:

FROM `fh-bigquery.wikipedia_v2.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
1.7s elapsed, 180 GB processed

Ouch. 180 GB processed — but at least the date partitioning to cover only June worked. But the same in v3, our clustered table:

FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
1.8s elapsed, 112 MB processed

Yes!!! With a clustered table we can now do a SELECT * LIMIT 1, and it only scanned 0.06% of the data (in this case).

Base query: Barcelona, English, 180->10GB

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v2.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
AND wiki LIKE 'en'
AND title = 'Barcelona'
18.1s elapsed, 180 GB processed

Any query similar to this will incur costs of 180 GB on our v2 table. Meanwhile in v3:

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
AND wiki LIKE 'en'
AND title = 'Barcelona'
3.5s elapsed, 10.3 GB processed

On a clustered table we get the results on 1/6th of the time, for 5% of the costs.

Don’t forget the clusters order

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
-- AND wiki = 'en'
AND title = 'Barcelona'
3.7s elapsed, 114 GB processed

If I don’t make use of my of my clustering strategy (in this case: filtering by which of the multiples wikis first), then I won’t get all the possible efficiencies — but it’s still better than before.

Smaller clusters, less data

The English Wikipedia has way more data than the Albanian one. Check how the query behaves when we look only at that one:

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
AND wiki = 'sq'
AND title = 'Barcelona'
2.6s elapsed, 3.83 GB

Clusters can LIKE and REGEX

Let’s query for all the wikis that start with r, and for the titles that match Bar.*na:

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
AND wiki LIKE 'r%'
AND REGEXP_CONTAINS(title, '^Bar.*na')
4.8s elapsed, 14.3 GB processed

That’s nice behavior —clusters can work with LIKE and REGEX expressions, but only when looking for prefixes.

If I look for suffixes instead:

SELECT wiki, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
AND wiki LIKE '%m'
AND REGEXP_CONTAINS(title, '.*celona')
(5.1s elapsed, 180 GB processed

That takes us back to 180 GB — this query is not using clusters at all.


This query will scan all data within the time range, regardless of what table I use:

SELECT wiki, title, SUM(views) views
FROM `fh-bigquery.wikipedia_v2.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
GROUP BY wiki, title
64.8s elapsed, 180 GB processed

That’s a pretty impressive query — we are grouping thru 185 million combinations of (wiki, title) and picking the top 10. Not a simple task for most databases, but here it looks like it was. Can clustered tables improve on this?

SELECT wiki, title, SUM(views) views
FROM `fh-bigquery.wikipedia_v3.pageviews_2017`
WHERE DATE(datehour) BETWEEN '2017-06-01' AND '2017-06-30'
GROUP BY wiki, title
22.1 elapsed, 180 GB processed

As you can see here, with v3 the query took one third of the time: As the data is clustered, way less shuffling is needed to group all combinations before ranking them.


My data can’t be date partitioned, how do I use clustering?

Two alternatives:
1. Use ingestion time partitioned table with clustering on the fields of your interest. This is the preferred mechanism if you have > ~10GB of data per day.
2. If you have smaller amounts of data per day, use a column partitioned table with clustering, partitioned on a “fake” date optional column. Just use the value NULL for it (or leave it unspecified, and BigQuery will assume it is NULL). Specify the clustering columns of interest.

We discussed clustering on an id without date partitioning on Stack Overflow.

What we learned today

  • Cluster your tables — it’s free and it can optimize many of your queries.
  • The main cost of clustering is longer load times, but it’s worth it.
  • Make a sensible decision of what columns you’ll cluster by and the order of them. Think about how people will query your tables.
  • Clustered tables are in beta — check the docs for limitations and updates.
  • Engineers tip: I’ve heard that for now it’s better if you load data into your clustered tables once per day. On that tip, I’ll keep loading the hourly updates to v2, and insert the results into v3 once the day is over. From the docs: “Over time, as more and more operations modify a table, the degree to which the data is sorted begins to weaken, and the table becomes partially sorted”.

Next steps

Clustered tables in only one of many new BigQuery capabilities: Check out GIS support (geographical), and BQML capabilities (machine learning within BigQuery).

Want more stories? Check my Medium, follow me on twitter, and subscribe to reddit.com/r/bigquery. And try BigQuery — every month you get a full terabyte of analysis for free.Counting uniques faster in BigQuery with HyperLogLog++
As a data exploration task, counting unique users is usually slow and resource intensive. This is because your database…medium.freecodecamp.org

How to train and predict regression and classification ML models using only SQL — using BigQuery ML


In my book (Data Science on the Google Cloud Platform), I walk through a flight-delay prediction problem and show how to address it using a variety of tools including Spark Mlib and TensorFlow. Now that BigQuery ML has been announced, I thought I’d show how to predict flight delays using BQ ML.

Predict whether this flight will arrive late. Using only SQL.

Make no mistake — you still have to collect the data, explore it, clean it up, and enrich it. Essentially all the stuff I do in Chapter 1–9. In Chapter 10, I used TensorFlow. In this article, I will use BQML.

Create Regression Model

Here’s a BigQuery query to create the model:

CREATE OR REPLACE MODEL flights.arrdelay
(model_type='linear_reg', input_label_cols=['arr_delay']) AS
arr_delay IS NOT NULL

Note that:

  1. It starts with a CREATE MODEL, with the name of the model looking just like a table name. Note: ‘flights’ is the name of the dataset to store the resulting model — so, you’ll need to create an empty dataset before running the query.
  2. The options specify the algorithm — in this case, a linear regression algorithm, with arr_delay being the label
  3. Essentially, I’m pulling in the predictor and label variables in the SELECT

About 10 minutes later, the model is trained and evaluation results have been populated for each iteration:

The loss here is mean squared error, so the model converges on iteration #6 with a RMSE of about sqrt(97) = 10 minutes.

Predict using Model

The purpose of training a model is to predict with it. You can do model predictions with a SQL statement:

SELECT * FROM ML.PREDICT(MODEL flights.arrdelay,
arr_delay AS actual_arr_delay
arr_delay IS NOT NULL
LIMIT 10))

This results in:

As you can see, because we trained the model to predict a variable called “arr_delay”, ML.PREDICT creates a result column named predicted_arr_delay. In this case, I’m pulling 10 rows from the original table and predicting the arrival delay for those flights.

Create Classification Model

In the book, I don’t actually try to predict the arrival delay as such. Instead, I predict the probability that a flight will be more than 15 minutes late. This is a classification problem, and you can do that by changing the training query slightly:

(model_type='logistic_reg', input_label_cols=['on_time']) AS
IF(arr_delay < 15, 1, 0) AS on_time,
arr_delay IS NOT NULL

Here’s the evaluation results:

and an example of the predictions:

Evaluating the model

It is possible to evaluate the model on an independent dataset. I don’t have one handy, so I’ll just show you how to run the evaluation on the same dataset the model was trained on:

IF(arr_delay < 15, 1, 0) AS on_time,
arr_delay IS NOT NULL

The result:

BQML is really easy and really powerful. Enjoy!

What Facebook’s Cryptocurrency Means: 6 Predictions


Discussions about Facebook’s data privacy and corporate power are about to extend to money. Grab the popcorn!

Photo: Glen Carrie/Unsplash

By Caitlin Long

What does Facebook’s new cryptocurrency mean? According to news reports, Facebook will shortly launch Libra, a global cryptocurrency available to users of its suite of platforms (including Messenger and WhatsApp). Presumably any merchant with an account on these platforms could transact in the cryptocurrency with customers who also have accounts — for anything, such as online purchases, and physical-world purchases such as groceries and restaurants. Facebook this week revealed plans to announce details on June 18 and confirmed its cryptocurrency will be a “stablecoin” whose value will be tied to a basket of fiat currencies. Based on Facebook’s statement and several anonymous comments made by people tied to the project in interviews with The Information, here are my six predictions.

1. Facebook’s cryptocurrency will be a powerful force for good in developing countries, which is where Facebook intends to market the product

Why? Because central banks in developing countries are notorious for their lack of discipline in maintaining the value of their fiat currencies, which too often lose purchasing power. The best example among many is Venezuela, which is experiencing hyperinflation worse than that of Germany after World War I. By providing citizens of developing nations with access to a store-of-value that is more reliable than their government-backed currencies, Facebook’s cryptocurrency will indirectly exert fiscal and monetary discipline on developing nations — which will improve the lives of many people globally.

2. Facebook will pay interest to holders of its cryptocurrency, and this will eventually lead to populist calls to repeal corporate subsidies to banks at the heart of the US banking system

I’ll go out on a limb and predict Facebook will pay interest to users of its cryptocurrency. Why? Because the assets backing the cryptocurrency will generate interest income (especially if, according to some reports, that basket includes “low-risk securities”). If Facebook doesn’t share these interest spoils with users, a chorus of critics will loudly publicize how much money Facebook and its partners are pocketing.

What’s the magnitude of the interest income at play? If Facebook parks the entire US dollar balance at the Federal Reserve via one of its bank partners, for example, it could earn 2.35% risk-free — that’s $235 million for every $10 billion deposited into its cryptocurrency. These profits will quickly turn into a new hot potato for Facebook politically, if not shared with users.

But there’s a side benefit — the brouhaha this issue could create would reveal the magnitude of corporate welfare at the heart of the US banking system. The 2.35% number is the actual interest rate the Fed pays its member banks for interest on excess reserves (IOER) — and this year it’s projected to amount to $36 billion of corporate welfare paid to US banks, which equates to roughly half the amount the US spends on its food stamp program. Just imagine how critics will have a field day shouting “corporate welfare for Facebook” if Facebook and its partners simply pocket that amount.

It’s true that other stablecoin issuers almost always pocket the float rather than sharing it with their customers. But Facebook’s stablecoin will probably be too big and visible to get away with this — so it’s unlikely to be able to sweep the issue under the rug. That’s a good segue to the next point.

3. Facebook’s foundation will grow to garner big power in global capital markets

Facebook plans to cede governance control of its project to an independent foundation, which it recently formed in Switzerland. This is a positive — not only does it give Facebook defense against antitrust allegations, but it also helps reduce the degree of its cryptocurrency’s centralization. This foundation is likely to become a huge power within global capital markets relatively quickly — because it will do what central banks do, which is to define the basket weights for the fiat currencies to which the stablecoin is pegged and manage the assets to ensure the peg doesn’t break. There are plenty of powerful “basket-setters” in capital markets, and their power to move markets can be significant — think of the committee that defines components of the Dow Jones Industrial Average Index (DJIA) or the S&P 500 Index, or central banks that peg their currencies to baskets (such as China’s PBOC).

Along these lines, at first blush it struck me as backwards that Facebook is selling the right to participate in its network for $10 million each, since usually the transaction processers (“miners”) in cryptocurrency markets are paid for their services rather than need to pay-to-play. But, remember #2 — there’s a big pot of interest income for all of them to divide among themselves, especially if they don’t pay interest to users, and a lot of foreign exchange trading volume is at stake. No wonder why dozens of banks will potentially be involved.

4. Facebook will face regulatory uncertainty — which will shed light on many outdated financial regulations in the process

Is Facebook’s cryptocurrency a security? If it is, will users face the absurdity of needing a US brokerage account to buy a cup of coffee with it? Will Facebook catch breaks from regulators that smaller start-ups haven’t — because of the tax data honeypot Facebook’s project will generate for governments? That’s a good segue to the next prediction.

5. Facebook’s regulatory reporting program will open all kinds of interesting discussions

We may be about to find out how many of Facebook’s 2.3 billion users are real — since users of its cryptocurrency would need to prove their identity and pass know-your-customer compliance requirements. The Information story noted Facebook “plans to provide more stringent forms of identity verification and fraud detection than do most cryptocurrencies.”

But there’s more. Discussions about Facebook’s data privacy and corporate power are about to extend to money. Grab the popcorn!

This will open all kinds of conversations about the extent of data privacy, financial privacy, overseas asset reporting and tax compliance and reporting burdens — and could potentially challenge the extra-territorial reporting requirements imposed by the US government on non-US businesses. Governments everywhere will view Facebook’s cryptocurrency as a huge honeypot of data about how users spend money — with all the privacy and tax reporting implications that data honeypot entails, because every transaction would be traceable by governments. This would give certain officials what they’re hoping for, which is the ability to trace, monitor and analyze every dollar spent rather than piecing together piecemeal reports (currently, banks file suspicious activity reports for transactions above $10,000). Here’s one example in a statement made during a May 2019 speech by Sigal Mandelker, Under Secretary for Terrorism and Financial Intelligence:

When FinCEN analyzed millions of dollars of remittance transactions with suspected links to terrorism, it found they averaged less than $600 each. In an era where a radicalized suicide bomber can bring a tragic end to the lives of hundreds for nothing more than the price of duct tape, a vest, and supplies, we cannot afford to allow any money to flow to terrorists.(emphasis added)

Moreover, these reporting requirements are about to grow substantially once the Financial Action Task Force recommends that all financial transactions (including cryptocurrencies) embed data about the beneficial owner in each transaction, for both the sender and receiver.

As I’ve explained in a previous Forbes.com post, I doubt the data dragnet required by US reporting requirements would survive a constitutional challenge. How interesting if Facebook were at the center of such a challenge. Would it fight or cooperate??

6. Facebook’s cryptocurrency will turn out, in the end, to be a Trojan horse that benefits bitcoin

Here’s my biggest prediction: Facebook’s foray into cryptocurrency will end up benefiting bitcoin. It will take time, but Facebook will greatly accelerate the pace of teaching people about cryptocurrencies. And when this happens, more people will turn to bitcoin for one simple reason — bitcoin is scarce, while Facebook’s cryptocurrency is not. People will migrate over time to the most honest ledger for storing their hard-earned wealth — and that’s not fiat currencies or derivatives thereof, including Facebook’s cryptocurrency.

This phenomenon actually happened in Venezuela, as early bitcoiner Nick Spanos recently pointed out to me. When the Maduro regime introduced the ill-fated petro cryptocurrency, the government made a concerted effort to educate Venezuelans about cryptocurrencies — and it correlated to a spike in bitcoin use by Venezuelans.

Facebook’s foray into cryptocurrency will likely end up being a beneficial detour on the path to broader bitcoin adoption. Bring it on!

Caitlin Long is a 22-year Wall Street veteran who has been active in bitcoin since 2012, and whose passion is a fair and stable financial system.

BlueMango 蓝芒频道